LoRA and QLoRA
On this page
Fully fine-tuning a 7B model with Adam needs well over 100 GB of GPU memory before counting activations: the weights, a gradient for every weight, and two optimizer values plus a 32-bit master copy for every weight. That means several data-centre GPUs for one experiment, and a full 14 GB copy of the model for every task you train. LoRA trains a tiny add-on instead and leaves the original weights untouched; QLoRA also stores those frozen weights in 4 bits, so the same job fits on one consumer-class GPU.
How it works

LoRA: learn a low-rank update
For a weight matrix W (say 4096 × 4096), full fine-tuning learns a change ΔW of the same size. LoRA assumes the useful change has low rank and writes it as the product of two thin matrices: ΔW = A·B, with A of size 4096 × r and B of size r × 4096, where the rank r is small (4 to 64). The layer computes y = x·W + (α/r)·x·A·B.
- W is frozen. Only A and B get gradients and optimizer state. For r = 8 that is 65,536 trainable values instead of 16.8 million for this layer, about 0.4%.
- B starts at zero, so training starts from exactly the base model.
- α/r is a scale that keeps the effective learning rate steady when you change r.
- Where: usually the attention projections (query, key, value, output), often the MLP layers too. Adapting more layers usually helps quality at small cost.
After training you can merge (W' = W + (α/r)·A·B) for zero extra inference cost, or keep adapters separate: one base model in memory, many small adapter files (tens of MB each), swapped per customer or task.
QLoRA: same idea, 4-bit frozen base

- The base weights are stored in a 4-bit format designed for normally distributed weights (NF4). A 7B model drops from 14 GB to about 4 GB.
- During the forward and backward pass, each layer is dequantised to BF16 on the fly; gradients flow through it into A and B, which stay in BF16.
- Two memory tricks: the quantisation scales are quantised too ("double quantisation"), and optimizer state can be paged to CPU memory when the GPU runs short.
- The QLoRA paper reported quality close to 16-bit fine-tuning on its benchmarks; in practice, check on your own evals.
Trade-offs and costs
- Memory: LoRA removes most gradient and optimizer memory; QLoRA also shrinks the weights. Activations still scale with batch size and sequence length.
- Speed: QLoRA is slower per step than LoRA because of the dequantisation work; LoRA is roughly as fast as a forward and backward pass of the frozen model.
- Quality: for style, format and narrow tasks, LoRA usually comes close to full fine-tuning. For large shifts (a new language, lots of new domain knowledge), full fine-tuning or continued pretraining can still win.
- Rank: higher r gives more capacity and more memory; past a point it stops helping. r = 8–16 is a common start.
- Serving: merged adapters cost nothing at inference; unmerged adapters add a small matrix multiply but let one GPU serve many tasks.
In production
- Libraries such as Hugging Face PEFT wrap existing layers; you choose the target modules, r, α and dropout.
- Multi-adapter serving: inference engines can batch requests that use different LoRA adapters on the same base model, which makes per-customer fine-tunes affordable.
- Version adapters with their base model. An adapter trained on one base model is useless on another, even a minor version change.
- Merging into a quantised model needs care: dequantise, merge, re-quantise, and re-evaluate.
When to reach for it
- Almost every fine-tune of a model you host yourself, unless you have the budget and a reason for full fine-tuning.
- QLoRA when GPU memory is the constraint: a single 24–48 GB GPU, or a large model (70B) on one or two GPUs.
- Many small task- or customer-specific variants of the same base model.
Example
A team wants a 13B open model to write SQL in their warehouse's dialect, trained on 8,000 question–query pairs. Full fine-tuning would need a multi-GPU node. With LoRA (r = 16 on all attention and MLP projections) it trains on one 80 GB GPU; with QLoRA it fits on a 48 GB card. On the 300-question eval set the LoRA model reaches the accuracy target. They keep the adapter separate because a second team trains its own adapter on the same base, and both are served from one deployment.
Common mistakes
- Initialising both A and B randomly (the model changes before training) or both at zero (nothing learns).
- Forgetting the α/r scale when merging, so the merged model doesn't match the evaluated one.
- Adapting only the query and value projections by habit, when adapting all linear layers is often better for little extra cost.
- Expecting LoRA to add lots of knowledge. It is good at behaviour; facts belong in retrieval.
- Mixing adapters and base versions.
Say this in the interview
"LoRA freezes the pretrained weights and learns each layer's update as a low-rank product A·B, with B initialised to zero so training starts from the base model. Only a fraction of a percent of parameters are trained, so gradients and optimizer state nearly disappear, and adapters are small enough to keep one per task. After training I can merge them for zero latency or serve many on one base. QLoRA adds a 4-bit frozen base, dequantised per layer, so a 70B model can be fine-tuned on one or two GPUs at some speed cost."
Interview questions on this
- Where does full fine-tuning's memory go? Weights, gradients, and Adam's two moments plus an FP32 master copy: about 16 bytes per parameter with mixed precision.
- Why initialise B to zero? So the adapter is a no-op at step 0; A must still be random so gradients reach B.
- What does r control? The capacity of the update, the trainable parameter count and memory.
- Does LoRA slow down inference? Not after merging; unmerged adapters add a small extra multiply.
- QLoRA vs LoRA? Same adapters; QLoRA stores the frozen base in 4-bit to save memory, at the cost of slower steps.
- When would you choose full fine-tuning? Large domain or language shifts, enough compute, and evidence that LoRA plateaus below target.
Used in
- AI Engineering: Implement a LoRA Linear Layer; Fine-tune, RAG or Prompt?; Supervised Fine-tuning; Quantization: INT8, INT4 and FP8; Training Memory Math.
- AI System Design: Design a Fine-tuning Platform.