Checkpoints and Fault Tolerance in Long Training Runs

Updated Nov 16, 2026 · 5 min read
On this page

    A 30-day pretraining run on 2,048 GPUs saves a checkpoint every 3 hours, and each save pauses training for 4 minutes. On day 12 one GPU throws a memory error at 2:50 into a 3-hour window. The whole job stops, because every GPU waits on every other in a synchronous step. Restarting takes 25 minutes, and almost three hours of work across 2,048 GPUs is gone. At that scale failures are not rare events but a steady rate: fault tolerance is the engineering of losing as little as possible each time, and checkpointing is the core of it.

    How it works

    Why big jobs fail often

    A failure throws away everything since the last checkpoint, plus the time to detect, restart and reload.
    A failure throws away everything since the last checkpoint, plus the time to detect, restart and reload.

    Synchronous training is only as reliable as its least reliable part. If one GPU fails on average once every few years, 16,000 of them fail every few hours. Meta's Llama 3 paper reported 419 unexpected interruptions in 54 days of pretraining on 16,384 GPUs, about one every 3 hours, most of them hardware (GPU and HBM faults leading). Add network flaps, storage hiccups, software bugs and silent stalls (a hung collective where nothing crashes but nothing progresses).

    The cost of each failure is: work since the last checkpoint + time to detect + time to get replacement nodes + time to reload the checkpoint and warm up. The metric to maximise is goodput: useful training time divided by wall-clock time.

    What goes into a checkpoint

    To resume exactly, not just to have model weights:

    • Model weights (usually sharded, one file set per rank).
    • Optimizer state: with Adam, three quarters of the bytes (see Training Memory Math).
    • Learning-rate scheduler, step count, gradient scaler state.
    • Data loader position and every random number generator state, so the resumed run sees the same data in the same order.
    • Config and code version.

    For a 70B model with Adam that's about 1.12 TB per checkpoint: 112 s at 10 GB/s of storage bandwidth, 11 s at 100 GB/s. Every rank writes its own shard in parallel, so aggregate bandwidth is what matters.

    How often

    The checkpoint interval balances time spent saving against work lost per failure; cheaper saves and faster restarts both cut waste.
    The checkpoint interval balances time spent saving against work lost per failure; cheaper saves and faster restarts both cut waste.

    A classic first-order answer (Young and Daly): interval ≈ √(2 × save cost × mean time between failures). Saving too often wastes time saving; too rarely loses more work per failure. With a failure every 3 hours (ckpt_math.py):

    • Hourly blocking checkpoints of 60 s with a 15-minute restart waste about 27% of the run.
    • The optimal interval for that save cost (19 min) brings it to about 19%.
    • Making saves asynchronous (5 s pause) allows 5.5-minute intervals: about 11%.
    • Cutting restart to 5 minutes with hot spare nodes: under 6%.

    The lesson: reduce the save stall and the restart time, then checkpoint often.

    Saving without stopping

    Copy state to CPU memory quickly, then upload it in the background while training continues; every rank writes its own shard.
    Copy state to CPU memory quickly, then upload it in the background while training continues; every rank writes its own shard.
    1. Snapshot the state from GPU to pinned CPU memory: seconds, the only pause.
    2. Persist from CPU to local NVMe and then object storage in background threads while training continues.
    3. Commit: write a "complete" marker only after every shard has landed, so a crash mid-save never leaves a half checkpoint looking valid.
    4. Retain the last few checkpoints (a bad one can exist: corrupted, or saved after a loss spike) and keep some long-term for evaluation.

    Some systems also keep an in-memory copy of each rank's shard on a peer node, so a single node loss recovers from RAM instead of storage.

    Recovering fast

    • Detection: health checks, NCCL/collective timeouts, heartbeat monitors, and watching for steps that stop progressing.
    • Replacement: a pool of hot spare nodes to swap in, instead of waiting for new capacity.
    • Automatic restart: the job launcher (e.g. torchrun elastic, a Kubernetes operator, Slurm requeue) restarts from the newest complete checkpoint without a human.
    • Quarantine: run diagnostics on the failed node and keep it out until it passes; flaky hardware tends to fail again.

    Trade-offs and costs

    • Storage cost and bandwidth: frequent 1 TB checkpoints add up; keep few, delete old ones, keep weights-only copies for evaluation.
    • Async complexity: CPU memory for the snapshot, and care so the next step doesn't modify tensors mid-copy.
    • Reshard on resume: resuming on a different GPU count needs checkpoint formats that can be resharded (PyTorch Distributed Checkpoint, Megatron's distributed checkpoint).
    • Silent data corruption: some hardware errors produce wrong numbers without crashing; watch for loss or gradient-norm anomalies and be ready to roll back further.

    In production

    • Track goodput, failures per day by cause, mean time to detect and to restart, and checkpoint save stall.
    • Test restore regularly: a checkpoint you haven't loaded is a hope, not a backup.
    • Make resume deterministic: same data order, same RNG, so a restarted run continues the same loss curve.
    • For fine-tuning jobs (hours, few GPUs), the same ideas at a smaller scale: save every N steps to object storage and auto-resume preemptible instances.

    When to reach for it

    Any training job long or large enough that a crash costs real money: multi-node pretraining, long fine-tunes, and anything on preemptible/spot GPUs. It's also a standard follow-up in ML infra and "design a training platform" interviews.

    Example

    A 2,048-GPU run sees about 8 interruptions per day. Baseline: hourly blocking saves of 4 minutes and 25-minute manual restarts; by the formula above, about 37% of GPU time is wasted. Changes: async sharded checkpoints every 15 minutes (8 s stall), automatic restart with 16 hot spare nodes (restart 6 minutes), node health checks before rejoining, last 5 checkpoints kept. Waste drops to about 8%, which over a 30-day run is roughly 9 days of the full cluster saved. (Example numbers.)

    Common mistakes

    • Saving only the weights, then being unable to resume optimizer state, schedule or data position.
    • Checkpointing on a fixed long interval without doing the failure-rate math.
    • Never testing restore.
    • Writing checkpoints through one rank instead of all ranks in parallel.
    • No "complete" marker, so a crash during save corrupts the latest checkpoint.
    • Ignoring hangs: a stuck collective with no timeout can waste hours without any error.

    Say this in the interview

    "In synchronous training one failed GPU stops the job, and at thousands of GPUs you see failures every few hours, so I'd optimise goodput. A checkpoint has to hold weights, optimizer state, scheduler, data loader position and RNG state, sharded and written by every rank in parallel. I'd make saves asynchronous (a quick copy to CPU memory, then background upload with a completion marker), choose the interval with √(2 × save cost × MTBF), and cut restart time with health checks, collective timeouts, hot spare nodes and automatic resume from the latest complete checkpoint."

    Interview questions on this

    1. What must a checkpoint contain to resume exactly? Weights, optimizer state, scheduler and step, scaler, data position, RNG states.
    2. How do you pick the checkpoint interval? Young/Daly: √(2 × save cost × MTBF), then adjust for restart time.
    3. How do you checkpoint 1 TB without pausing for minutes? Sharded parallel writes plus an async copy to CPU memory and background upload.
    4. A run hangs without crashing. How do you catch it? Collective timeouts, step-progress watchdogs, heartbeats.
    5. How do you resume on a different number of GPUs? A reshardable checkpoint format and deterministic data indexing.
    6. What's goodput and how would you raise it? Useful time / wall time; shorter stalls, faster detection and restart, fewer repeat failures.

    Used in

    • AI Engineering: Parallelism for Training; ZeRO and FSDP; Training Memory Math; GPU Job Scheduling and Queues; Your Training Loss Spiked: How Do You Find Out Why?
    • AI System Design: Design a Fine-tuning Platform.