How to optimize LoRA adapters for fine-tuning LLMs on consumer hardware?

All answers to this question.

For a 24GB card, you absolutely need to use QLoRA (Quantized LoRA) with 4-bit loading. Set your rank (r) to 16 and your alpha to 32 as a starting point. In my projects last year, I found that increasing the rank beyond 64 yielded diminishing returns while significantly increasing the memory footprint. Also, make sure you are only targeting the linear layers like 'q_proj' and 'v_proj'. If you try to train all modules, you will definitely hit that OOM error. Using the Unsloth library can also give you a 2x speedup and significantly lower VRAM usage for Llama models.

   Answered 2024-05-13 by Robert Chen

Have you monitored your gradient accumulation steps? Sometimes the OOM isn't from the model itself but from a batch size that is too high for the optimizer states. Are you using the standard AdamW optimizer, or have you tried a more memory-efficient version like 8-bit Adam or the newer paged optimizers?

   Answered 2024-05-15 by Kevin Douglas

• Kevin, I was using a batch size of 4 with AdamW. After your comment, I switched to 8-bit Adam and reduced the batch size to 1 while increasing gradient accumulation to 4. This stabilized the VRAM at 21GB, leaving enough overhead for the system. It’s a bit slower per iteration, but at least the training doesn’t crash halfway through the first epoch now. Thanks for the tip!

     Commented 2024-05-16 by Amanda Roberts

Don't forget to enable Gradient Checkpointing. It trades a bit of compute time to save a massive amount of VRAM by not storing all intermediate activations during the forward pass.

   Answered 2024-05-17 by Laura White

• Laura is spot on. Gradient Checkpointing is the single most effective "free" memory hack for anyone training on a single-GPU setup.

     Commented 2024-05-18 by Robert Chen