Your Training Loss Spiked: How Do You Find Out Why?

Free with account Updated Nov 16, 2026 · 3 min read
On this page

    Problem statement

    "We're three days into a training run on 64 GPUs. Overnight the loss jumped from 2.1 to 6.8 at step 41,200. What do you do?"

    A debugging question for training and infra roles. The interviewer wants a calm order of operations: protect the run, classify the spike, find the cause from logs and data, then fix it so it doesn't come back.

    Why it happens

    The same spike can end three ways: it recovers, it leaves the run worse off, or it diverges.
    The same spike can end three ways: it recovers, it leaves the run worse off, or it diverges.
    • Bad data: a shard of near-duplicate pages, binary junk, a very long sequence, or a batch from one narrow source pushes one large, wrong update.
    • Learning rate too high for this moment: spikes often appear right after warm-up ends or when the batch size changes.
    • Optimizer state: Adam divides by a running estimate of squared gradients; after a long quiet stretch that estimate is tiny, so one large gradient produces a huge update.
    • Numerics: FP16 overflows past about 65,000; attention logits or output logits can grow until softmax saturates. Large models are especially prone to this.
    • System faults: a checkpoint restored without its optimizer state or data position, a misconfigured node, a silent hardware error that corrupts a gradient.