Training Memory Math: Weights, Gradients, Optimizer, Activations
On this page
An engineer loads a 7B model for inference on a 24 GB GPU without trouble: 14 GB of BF16 weights. They start a full fine-tune on an 80 GB GPU and it crashes with out-of-memory before the first step finishes. Nothing is wrong; training simply needs about 8× more memory for the model state than inference, plus activations that grow with sequence length and batch size. Being able to do this arithmetic quickly is one of the most common AI infra interview checks.
How it works

The four buckets
With mixed precision (BF16 compute) and the Adam optimizer, per parameter:
| What | Bytes per parameter | Why |
|---|---|---|
| Weights | 2 | BF16 copy used in forward and backward |
| Gradients | 2 | one per weight, BF16 |
| Optimizer state | 12 | FP32 master weights (4) + Adam momentum (4) + variance (4) |
| Total model state | 16 | before activations |
Inference needs only the 2-byte weights (plus KV cache), so training state is 8× larger. For a 7B model (6.7B parameters): 13.4 + 13.4 + 80.4 = 107 GB, which already exceeds one 80 GB GPU.
Some setups differ: pure BF16 training without FP32 master weights uses less (with some risk to accuracy); FP32 gradients use more; SGD with momentum stores 4 bytes instead of 8 for its state. Adjust the table, keep the method.
Activations

The forward pass saves intermediate results so the backward pass can compute gradients. A widely used estimate (Korthikanti et al., 2022) for one transformer layer in 16-bit, no tensor parallelism:
activations per layer ≈ s × b × h × (34 + 5 × a × s / h) bytes
where s is sequence length, b micro-batch size, h hidden size and a the number of attention heads. The second term is the attention score matrix, which grows with s². For the 7B model (32 layers, h = 4,096, 32 heads) at s = 4,096, b = 1:
- With naive attention: about 104 GB.
- With fused attention (FlashAttention never stores the scores): about 18 GB.
- With activation checkpointing (keep only each block's input, recompute the rest during backward): about 1 GB, for roughly 30% more compute.
Activations scale linearly with batch size and with sequence length (quadratically with naive attention), which is why long-context training is mostly an activation problem.
Putting numbers on it
The included memory_calc.py reproduces every number in this article:
7B, 1 GPU, flash attention total 125.5 GB
7B, 8 GPUs ZeRO-3, flash total 31.7 GB
7B, 8 GPUs ZeRO-3, flash, checkpointing total 14.5 GB
13B, 8 GPUs ZeRO-3, flash, mb=2 total 83.0 GB
13B, 8 GPUs ZeRO-3, flash, mb=2, ckpt total 29.4 GB
7B LoRA: frozen weights 13.4 GB + trainable state 0.54 GB
Add a few GB for the framework, CUDA context, temporary buffers and memory fragmentation; plan on 10–20% headroom.
Trade-offs and costs

- Sharding (ZeRO/FSDP) divides model state by the number of GPUs at the cost of communication.
- Activation checkpointing trades about a third more compute for most of the activation memory; selective checkpointing (only the expensive parts) is a middle ground.
- LoRA/QLoRA freezes the base weights, so gradients and optimizer state exist only for small adapters; QLoRA also stores the frozen base in 4 bits.
- 8-bit optimizers shrink Adam's two moment tensors from 8 to about 2 bytes per parameter.
- Offloading to CPU or NVMe fits almost anything, slowly.
- Gradient accumulation keeps the effective batch size while cutting the micro-batch, so activations shrink.
In production
- Do the arithmetic before booking GPUs; then confirm with a short run and the framework's memory profiler (
torch.cuda.max_memory_allocated, memory snapshots). - Watch for fragmentation: allocated memory well below reserved memory means the allocator is wasting space; allocator settings or fixed shapes help.
- Long-context fine-tuning: start with fused attention and checkpointing; add sequence or context parallelism when a single sequence's activations won't fit.
When to reach for it
Every time someone asks "how many GPUs do we need to train or fine-tune this?", when a run fails with out-of-memory, and when choosing between full fine-tuning and LoRA. It's also the basis for estimating cost.
Example
Interview prompt: "Can we fully fine-tune a 13B model on one server of 8 × 80 GB at 4,096 tokens?" Model state: 13B × 16 = 208 GB, so plain data parallelism (208 GB per GPU) fails. With FSDP across 8 GPUs: 26 GB each. Activations with fused attention, micro-batch 2: about 57 GB per GPU, total about 83 GB: just over the limit. Turn on activation checkpointing: activations drop to about 3.4 GB, total about 30 GB, leaving room. Answer: yes, with FSDP plus checkpointing; or use LoRA and fit it on far less.
Common mistakes
- Counting only the weights (2 bytes per parameter) for training.
- Forgetting the FP32 master copy in mixed precision.
- Ignoring activations, or assuming they don't depend on sequence length.
- Assuming ZeRO shards activations. It doesn't; checkpointing and sequence parallelism do.
- No headroom for framework overhead and fragmentation.
Say this in the interview
"With mixed precision and Adam, training needs about 16 bytes per parameter: 2 for BF16 weights, 2 for gradients and 12 for the optimizer, meaning an FP32 master copy plus two Adam moments. So a 7B model needs about 110 GB before activations, versus 14 GB to serve it. Activations scale with sequence length times batch size; fused attention removes the quadratic score matrix and activation checkpointing cuts the rest for about 30% more compute. Then I'd shard the 16 bytes across GPUs with FSDP, or switch to LoRA so only adapters carry optimizer state, and keep 10–20% headroom."
Interview questions on this
- Memory to train a 70B model with Adam in mixed precision, excluding activations? 70B × 16 = 1.12 TB.
- Why does Adam need 12 bytes per parameter in mixed precision? FP32 master weights plus FP32 first and second moments.
- How does activation memory change if you double the sequence length? Doubles with fused attention; roughly quadruples the attention part without it.
- What does activation checkpointing cost? About one extra forward pass, roughly 30% more compute.
- Why can LoRA fine-tune a 7B model on a 24 GB GPU? Frozen weights in BF16 (or 4-bit) and optimizer state only for tiny adapters.
- Your run OOMs at step 1 but not during the forward pass. Why? The optimizer state is allocated on the first step, or the backward pass's gradients and temporary buffers push it over.
Used in
- AI Engineering: ZeRO and FSDP: Sharding Training State; Mixed Precision Training; Parallelism for Training; FlashAttention; LoRA and QLoRA; How Many GPUs Do You Need to Serve a 70B Model?
- AI System Design: Design a Fine-tuning Platform.