Your Training Loss Spiked: How Do You Find Out Why?
On this page
Problem statement
"We're three days into a training run on 64 GPUs. Overnight the loss jumped from 2.1 to 6.8 at step 41,200. What do you do?"
A debugging question for training and infra roles. The interviewer wants a calm order of operations: protect the run, classify the spike, find the cause from logs and data, then fix it so it doesn't come back.
Why it happens

- Bad data: a shard of near-duplicate pages, binary junk, a very long sequence, or a batch from one narrow source pushes one large, wrong update.
- Learning rate too high for this moment: spikes often appear right after warm-up ends or when the batch size changes.
- Optimizer state: Adam divides by a running estimate of squared gradients; after a long quiet stretch that estimate is tiny, so one large gradient produces a huge update.
- Numerics: FP16 overflows past about 65,000; attention logits or output logits can grow until softmax saturates. Large models are especially prone to this.
- System faults: a checkpoint restored without its optimizer state or data position, a misconfigured node, a silent hardware error that corrupts a gradient.
Try it yourself first
You have the full problem. Sketch your answer, then sign up free to read the complete walkthrough.
- The complete design, flow by flow, with pictures
- Deep dives into the hard parts
- What mid, senior and staff candidates are expected to say
- Full company question lists
Free forever. No card needed.