Mixed Precision Training

Updated Nov 16, 2026 · 5 min read
On this page

    A team switches a training run from FP32 to FP16 to get the GPU's tensor-core speed. Steps run about twice as fast, but after a few thousand of them the loss turns into NaN. They try BF16 instead: no NaNs, but the model learns visibly slower than the FP32 baseline. Both problems have the same cause, which is that 16 bits can't hold everything training needs, and the same standard fix: mixed precision, meaning do the heavy math in 16-bit and keep the values that must be exact (master weights, optimizer state, sensitive reductions) in 32-bit.

    How it works

    The formats

    How FP32, FP16, BF16 and FP8 split their bits between exponent (range) and mantissa (precision).
    How FP32, FP16, BF16 and FP8 split their bits between exponent (range) and mantissa (precision).
    • FP32: 8 exponent bits, 23 mantissa bits. Huge range, about 7 significant digits.
    • FP16: 5 exponent, 10 mantissa. About 3 digits, but the largest value is 65,504 and normal values stop at about 6 × 10⁻⁵ (subnormals reach 6 × 10⁻⁸ with lost precision). Gradients easily fall below that.
    • BF16 (bfloat16): 8 exponent, 7 mantissa. The same range as FP32, about 2–3 digits. Built for deep learning; the default for training on current GPUs and TPUs.
    • FP8 (E4M3 / E5M2): on Hopper-class and newer GPUs, used for the matrix multiplies in some large training runs, with per-tensor scaling.

    Tensor cores run 16-bit matrix multiplies several times faster than FP32 math (exact ratios depend on the GPU), and 16-bit tensors halve memory traffic, so the speed-up is real.

    The mixed-precision loop

    FP32 master weights are cast to BF16 for forward and backward; gradients update the FP32 copy.
    FP32 master weights are cast to BF16 for forward and backward; gradients update the FP32 copy.
    1. Keep FP32 master weights.
    2. Each step, cast them to BF16 (or FP16) and run forward and backward with 16-bit inputs. Matrix multiplies accumulate in FP32 inside the tensor cores.
    3. Ops that are sensitive to rounding (softmax, layer norm, loss, large sums) run in FP32. Frameworks decide this per operation (torch.autocast).
    4. Apply the optimizer update to the FP32 master weights with FP32 optimizer state.

    Memory cost: 2 bytes (16-bit weights) + 2 (gradients) + 4 (FP32 master) + 8 (Adam moments) = 16 bytes per parameter (see Training Memory Math). Mixed precision saves time and activation memory, not model-state memory.

    Why each piece is there

    FP16 loses tiny gradients unless the loss is scaled; 16-bit weights silently drop small updates unless a 32-bit master copy is kept.
    FP16 loses tiny gradients unless the loss is scaled; 16-bit weights silently drop small updates unless a 32-bit master copy is kept.

    The included precision_demo.py shows both failures:

    • Gradient underflow (FP16). A gradient of 1 × 10⁻⁹ becomes exactly 0 in FP16. Loss scaling multiplies the loss by a factor (say 1,024) before backward, so every gradient is scaled up into range, then divides it out before the update: the same gradient comes back as 9.9 × 10⁻¹⁰. Dynamic loss scaling raises the factor until an overflow (inf/NaN) appears, skips that step, halves the factor, and repeats. BF16's range makes this unnecessary.
    • Lost updates (16-bit weights). A weight of 1.0 updated by −3 × 10⁻⁵ stays exactly 1.0 in BF16, because the step is smaller than the gap between representable numbers near 1.0 (BF16 has only 8 bits of significand). After 1,000 such steps a BF16-only weight is still 1.0000; with an FP32 master copy it reaches the correct 0.9700. That's the "learns slower" symptom from the opening.
    • Overflow (FP16). 70,000 becomes infinity in FP16; activations or attention logits that large cause the NaNs.

    Trade-offs and costs

    • BF16 vs FP16: BF16 needs no loss scaling and rarely overflows; FP16 keeps more precision and is supported on older GPUs (pre-Ampere) that lack BF16.
    • Pure 16-bit training (no FP32 master) saves 4 bytes per parameter but needs tricks such as stochastic rounding or Kahan summation to avoid lost updates.
    • FP8 training speeds matrix multiplies further but needs careful scaling recipes and is mainly used for large pretraining runs by teams that can validate it.
    • Numerics differ from FP32, so results aren't bit-identical; compare loss curves, not exact values.

    In production

    • In PyTorch: torch.autocast(dtype=torch.bfloat16) around forward and loss, FP32 parameters in the optimizer, and GradScaler only for FP16. FSDP and DeepSpeed have mixed-precision settings for parameters, reductions and buffers.
    • Do gradient all-reduce in FP32 or BF16 (BF16 halves traffic; FP32 is safer for very large sums).
    • Watch for: NaN/inf in loss or gradient norm, loss-scale collapsing toward zero (FP16), training that matches FP32 early but drifts later.
    • For inference, the same model is usually served in BF16 or quantized further (see Quantization).

    When to reach for it

    Essentially always for training and fine-tuning on modern GPUs: BF16 mixed precision is the default. Reach for FP16 plus loss scaling only on hardware without BF16, and for FP8 only in large runs with a tested recipe.

    Example

    A 1.3B model trains at 0.42 s per step in FP32. With BF16 autocast and FP32 master weights it runs at 0.17 s per step and activation memory halves, so the micro-batch doubles. An earlier attempt to store weights in BF16 to save memory ("pure BF16") showed a loss curve 3% higher at 20,000 steps than the FP32 baseline; restoring FP32 master weights made the curves overlap. (Example numbers.)

    Common mistakes

    • Casting the whole model to FP16 and training (no master weights, no loss scaling).
    • Using FP16 without a gradient scaler, then chasing NaNs.
    • Running softmax or layer norm in 16-bit by hand, bypassing autocast's FP32 choices.
    • Expecting mixed precision to cut optimizer memory. It doesn't; it cuts time and activation memory.
    • Comparing BF16 and FP32 runs step by step instead of by loss curves and evals.

    Say this in the interview

    "Mixed precision runs forward and backward in 16-bit on tensor cores, which is several times faster and halves activation memory, but keeps FP32 master weights and optimizer state, so small updates aren't rounded away, and keeps sensitive ops like softmax and normalization in FP32. FP16 has a small range, so tiny gradients underflow and it needs dynamic loss scaling; BF16 has FP32's exponent range, so it doesn't, which is why BF16 is the default. Memory is still about 16 bytes per parameter with Adam."

    Interview questions on this

    1. Why keep an FP32 copy of the weights? Small updates are below 16-bit resolution near the weight's value and would be lost.
    2. What does loss scaling do and why doesn't BF16 need it? It shifts gradients into FP16's representable range; BF16 already has FP32's range.
    3. How does dynamic loss scaling handle an overflow? Skip the step, lower the scale, continue.
    4. Which operations should stay in FP32? Softmax, normalization, loss, large reductions, optimizer math.
    5. Does mixed precision reduce memory? Activations and temporary tensors yes; model state with Adam no.
    6. When would you consider FP8 training? Large runs on hardware with FP8 tensor cores, with a validated scaling recipe.

    Used in

    • AI Engineering: Training Memory Math; Quantization: INT8, INT4 and FP8; ZeRO and FSDP; GPU Basics for AI Engineers; Your Training Loss Spiked: How Do You Find Out Why?
    • AI System Design: Design a Fine-tuning Platform.