ZeRO and FSDP: Sharding Training State
On this page
A team fine-tunes a 7B model with full-parameter training on a server with eight 80 GB GPUs. The weights are only 14 GB, yet plain data parallelism runs out of memory before the first step. The reason: with Adam and mixed precision, each GPU holds 16 bytes per parameter (weights, gradients and three FP32 optimizer tensors), which is 112 GB, and all eight GPUs hold the same 112 GB. ZeRO and FSDP remove that duplication: each GPU stores only its share of the training state and borrows the rest from its peers for a moment when it needs it.
How it works
What a GPU holds during training
Per parameter, with BF16 compute and Adam:
- Weights: 2 bytes (BF16).
- Gradients: 2 bytes.
- Optimizer state: 12 bytes: an FP32 master copy of the weight (4), Adam's momentum (4) and variance (4).
That's 16 bytes per parameter, before activations (see Training Memory Math). Plain data parallelism replicates all of it on every GPU.
The three ZeRO stages

ZeRO (Zero Redundancy Optimizer, from Microsoft's DeepSpeed) shards progressively more of the state across the N data-parallel GPUs:
- ZeRO-1, shard the optimizer state. Each GPU updates only 1/N of the parameters, so it needs only 1/N of the optimizer state. 7B on 8 GPUs: 112 → 38.5 GB.
- ZeRO-2, also shard gradients. Gradients are reduce-scattered, so each GPU keeps only the summed slice it will update. → 26.25 GB.
- ZeRO-3, also shard the weights. Each GPU stores 1/N of every layer and gathers full layers only while computing them. → 14 GB, exactly 1/N of the original.
FSDP (Fully Sharded Data Parallel) is PyTorch's built-in version of ZeRO-3, with options that behave like stage 2.
One FSDP step

- Forward: before layer k runs, all-gather its weights from all GPUs; compute; free the gathered copy.
- Backward: all-gather layer k again, compute its gradients, free the weights.
- Reduce-scatter the gradients: each GPU ends up with the sum for its own slice.
- Update: each GPU updates its slice with its slice of optimizer state.
Peak memory is your shard plus a layer or two of full weights. Libraries prefetch the next layer's all-gather while the current layer computes, so communication overlaps with compute.
What it costs

Plain data parallelism already does an all-reduce of gradients, which is a reduce-scatter plus an all-gather. ZeRO-1 and 2 move about the same bytes. ZeRO-3 adds a second all-gather of the weights (forward and backward), so about 1.5× the traffic for up to N× less memory. On fast links with enough work per GPU the extra traffic hides behind compute; across slow networks it shows up as lower throughput.
Trade-offs and costs
- Memory vs communication: higher stages save more memory and talk more. Use the lowest stage that fits.
- Small batches hurt: with little compute per step, transfers can't hide.
- Hybrid sharding (HSDP / ZeRO++ style): shard within a server over NVLink and replicate across servers, cutting cross-node traffic at the cost of more memory.
- Offload: ZeRO-Offload / ZeRO-Infinity put optimizer state or weights in CPU memory or NVMe. Fits huge models on few GPUs, but steps get much slower.
- Checkpoints are sharded too, so saving and loading needs the right tooling (see Checkpoints and Fault Tolerance).
In production
- PyTorch FSDP (FSDP2 with DTensor in recent versions) and DeepSpeed ZeRO are the standard tools; Hugging Face Accelerate and most trainers expose them through config.
- Combine with activation checkpointing (recompute activations in backward) and mixed precision; for very large models, add tensor parallelism inside the server.
- Wrap at transformer-block granularity: each block is one unit that is gathered and freed.
- Measure step time and MFU before and after; if all-gather time is visible in the profile, tune prefetching, wrap size or switch to hybrid sharding.
When to reach for it
Full fine-tuning or pretraining when the training state doesn't fit on one GPU but the model would fit across a node or a few nodes: roughly 1B to 100B+ parameters. For LoRA fine-tuning the optimizer state is tiny, so sharding matters less; for the largest pretraining runs it is combined with tensor and pipeline parallelism.
Example
Full fine-tune of a 13B model on one server of 8 × 80 GB. Training state: 13B × 16 bytes = 208 GB, 26 GB per GPU with FSDP full sharding. Activations at sequence length 4,096 and micro-batch 2 would add about 55 GB per GPU without activation checkpointing (with fused attention; see Training Memory Math), so the team turns it on for every block (stored activations drop to about 3.4 GB, for about 30% more compute). Peak memory lands near 40 GB per GPU, and step time is within 10% of the theoretical compute time because all-gathers overlap with NVLink. (Example numbers.)
Common mistakes
- Thinking FSDP is model parallelism. Each GPU still runs the whole model on its own data; only storage is split.
- Jumping to ZeRO-3 when ZeRO-1 fits, paying extra communication for nothing.
- Forgetting activations, which ZeRO doesn't shard.
- Running FSDP across slow networks without hybrid sharding.
- Saving a sharded checkpoint and expecting a single-file model without consolidating it.
Say this in the interview
"With Adam and mixed precision, training state is about 16 bytes per parameter, and plain data parallelism duplicates it on every GPU. ZeRO shards it across the data-parallel group: stage 1 the optimizer state, stage 2 the gradients, stage 3 the weights too, so memory per GPU drops to roughly 16P/N. FSDP is PyTorch's ZeRO-3: each GPU owns a slice of each layer, all-gathers the full layer just before using it, frees it after, and reduce-scatters gradients. It costs about 1.5× the communication of data parallel, which overlaps with compute on fast links; across slow links I'd use hybrid sharding."
Interview questions on this
- How much memory per GPU for a 30B model with ZeRO-3 on 16 GPUs? 30B × 16 / 16 = 30 GB plus activations.
- Why does ZeRO-1 move no more data than plain data parallel? The all-reduce is already a reduce-scatter plus all-gather; ZeRO-1 just updates and gathers at different points.
- What does FSDP keep in memory at peak? Its shard plus the full weights of the layer or two being computed and prefetched.
- When would you use CPU offload? When GPU memory is the only blocker and step time matters less.
- What is hybrid sharding? Shard within a node, replicate across nodes, to keep heavy traffic on NVLink.
- FSDP vs tensor parallelism? FSDP splits storage but computes whole layers per GPU; TP splits the computation of each layer.
Used in
- AI Engineering: Parallelism for Training: Data, Tensor, Pipeline; Training Memory Math; Mixed Precision Training; GPU Interconnects and Collectives; Checkpoints and Fault Tolerance in Long Training Runs; LoRA and QLoRA.
- AI System Design: Design a Fine-tuning Platform.