Training Memory Math: Weights, Gradients, Optimizer, Activations

Updated Nov 16, 2026 · 5 min read
On this page

    An engineer loads a 7B model for inference on a 24 GB GPU without trouble: 14 GB of BF16 weights. They start a full fine-tune on an 80 GB GPU and it crashes with out-of-memory before the first step finishes. Nothing is wrong; training simply needs about 8× more memory for the model state than inference, plus activations that grow with sequence length and batch size. Being able to do this arithmetic quickly is one of the most common AI infra interview checks.

    How it works

    A full fine-tune of a 7B model at 4,096 tokens needs about 125 GB, most of it optimizer state.
    A full fine-tune of a 7B model at 4,096 tokens needs about 125 GB, most of it optimizer state.

    The four buckets

    With mixed precision (BF16 compute) and the Adam optimizer, per parameter:

    WhatBytes per parameterWhy
    Weights2BF16 copy used in forward and backward
    Gradients2one per weight, BF16
    Optimizer state12FP32 master weights (4) + Adam momentum (4) + variance (4)
    Total model state16before activations

    Inference needs only the 2-byte weights (plus KV cache), so training state is 8× larger. For a 7B model (6.7B parameters): 13.4 + 13.4 + 80.4 = 107 GB, which already exceeds one 80 GB GPU.

    Some setups differ: pure BF16 training without FP32 master weights uses less (with some risk to accuracy); FP32 gradients use more; SGD with momentum stores 4 bytes instead of 8 for its state. Adjust the table, keep the method.

    Activations

    Activation memory per layer depends on sequence length and batch; fused attention removes the quadratic term and checkpointing removes most of the rest.
    Activation memory per layer depends on sequence length and batch; fused attention removes the quadratic term and checkpointing removes most of the rest.

    The forward pass saves intermediate results so the backward pass can compute gradients. A widely used estimate (Korthikanti et al., 2022) for one transformer layer in 16-bit, no tensor parallelism:

    activations per layer ≈ s × b × h × (34 + 5 × a × s / h) bytes

    where s is sequence length, b micro-batch size, h hidden size and a the number of attention heads. The second term is the attention score matrix, which grows with s². For the 7B model (32 layers, h = 4,096, 32 heads) at s = 4,096, b = 1:

    • With naive attention: about 104 GB.
    • With fused attention (FlashAttention never stores the scores): about 18 GB.
    • With activation checkpointing (keep only each block's input, recompute the rest during backward): about 1 GB, for roughly 30% more compute.

    Activations scale linearly with batch size and with sequence length (quadratically with naive attention), which is why long-context training is mostly an activation problem.

    Putting numbers on it

    The included memory_calc.py reproduces every number in this article:

    7B, 1 GPU, flash attention                 total 125.5 GB
    7B, 8 GPUs ZeRO-3, flash                   total  31.7 GB
    7B, 8 GPUs ZeRO-3, flash, checkpointing    total  14.5 GB
    13B, 8 GPUs ZeRO-3, flash, mb=2            total  83.0 GB
    13B, 8 GPUs ZeRO-3, flash, mb=2, ckpt      total  29.4 GB
    7B LoRA: frozen weights 13.4 GB + trainable state 0.54 GB

    Add a few GB for the framework, CUDA context, temporary buffers and memory fragmentation; plan on 10–20% headroom.

    Trade-offs and costs

    Each memory-saving technique shrinks a different bucket at a different price.
    Each memory-saving technique shrinks a different bucket at a different price.
    • Sharding (ZeRO/FSDP) divides model state by the number of GPUs at the cost of communication.
    • Activation checkpointing trades about a third more compute for most of the activation memory; selective checkpointing (only the expensive parts) is a middle ground.
    • LoRA/QLoRA freezes the base weights, so gradients and optimizer state exist only for small adapters; QLoRA also stores the frozen base in 4 bits.
    • 8-bit optimizers shrink Adam's two moment tensors from 8 to about 2 bytes per parameter.
    • Offloading to CPU or NVMe fits almost anything, slowly.
    • Gradient accumulation keeps the effective batch size while cutting the micro-batch, so activations shrink.

    In production

    • Do the arithmetic before booking GPUs; then confirm with a short run and the framework's memory profiler (torch.cuda.max_memory_allocated, memory snapshots).
    • Watch for fragmentation: allocated memory well below reserved memory means the allocator is wasting space; allocator settings or fixed shapes help.
    • Long-context fine-tuning: start with fused attention and checkpointing; add sequence or context parallelism when a single sequence's activations won't fit.

    When to reach for it

    Every time someone asks "how many GPUs do we need to train or fine-tune this?", when a run fails with out-of-memory, and when choosing between full fine-tuning and LoRA. It's also the basis for estimating cost.

    Example

    Interview prompt: "Can we fully fine-tune a 13B model on one server of 8 × 80 GB at 4,096 tokens?" Model state: 13B × 16 = 208 GB, so plain data parallelism (208 GB per GPU) fails. With FSDP across 8 GPUs: 26 GB each. Activations with fused attention, micro-batch 2: about 57 GB per GPU, total about 83 GB: just over the limit. Turn on activation checkpointing: activations drop to about 3.4 GB, total about 30 GB, leaving room. Answer: yes, with FSDP plus checkpointing; or use LoRA and fit it on far less.

    Common mistakes

    • Counting only the weights (2 bytes per parameter) for training.
    • Forgetting the FP32 master copy in mixed precision.
    • Ignoring activations, or assuming they don't depend on sequence length.
    • Assuming ZeRO shards activations. It doesn't; checkpointing and sequence parallelism do.
    • No headroom for framework overhead and fragmentation.

    Say this in the interview

    "With mixed precision and Adam, training needs about 16 bytes per parameter: 2 for BF16 weights, 2 for gradients and 12 for the optimizer, meaning an FP32 master copy plus two Adam moments. So a 7B model needs about 110 GB before activations, versus 14 GB to serve it. Activations scale with sequence length times batch size; fused attention removes the quadratic score matrix and activation checkpointing cuts the rest for about 30% more compute. Then I'd shard the 16 bytes across GPUs with FSDP, or switch to LoRA so only adapters carry optimizer state, and keep 10–20% headroom."

    Interview questions on this

    1. Memory to train a 70B model with Adam in mixed precision, excluding activations? 70B × 16 = 1.12 TB.
    2. Why does Adam need 12 bytes per parameter in mixed precision? FP32 master weights plus FP32 first and second moments.
    3. How does activation memory change if you double the sequence length? Doubles with fused attention; roughly quadruples the attention part without it.
    4. What does activation checkpointing cost? About one extra forward pass, roughly 30% more compute.
    5. Why can LoRA fine-tune a 7B model on a 24 GB GPU? Frozen weights in BF16 (or 4-bit) and optimizer state only for tiny adapters.
    6. Your run OOMs at step 1 but not during the forward pass. Why? The optimizer state is allocated on the first step, or the backward pass's gradients and temporary buffers push it over.

    Used in

    • AI Engineering: ZeRO and FSDP: Sharding Training State; Mixed Precision Training; Parallelism for Training; FlashAttention; LoRA and QLoRA; How Many GPUs Do You Need to Serve a 70B Model?
    • AI System Design: Design a Fine-tuning Platform.