Parallelism for Training: Data, Tensor, Pipeline
On this page
A team wants to train a 70B model. The weights alone are 140 GB in BF16, and with gradients and Adam optimizer state, training needs about 16 bytes per parameter: over 1 TB before activations. No single GPU comes close. Even a 7B model, which fits, would take months on one GPU. Distributed training splits the work three basic ways (data, tensor and pipeline parallelism) and real training runs combine all three, matched to how fast the links between GPUs are.
How it works

Data parallelism (DP)
Every GPU holds a full copy of the model and processes a different slice of the batch. After the backward pass, GPUs all-reduce their gradients (sum them so everyone has the same average), and each applies the same update, so the copies stay identical.
- Scales throughput almost linearly while the model fits on one GPU.
- Communication: one all-reduce of all gradients per step. For a 7B model in BF16 that's 14 GB of gradients; a ring all-reduce sends about 2 × 14 GB per GPU per step, which is why it overlaps with the backward pass, layer by layer.
- Memory per GPU doesn't shrink. Sharded data parallelism (ZeRO, FSDP) fixes that by splitting optimizer state, gradients and weights across the data-parallel GPUs (see ZeRO and FSDP).
Tensor parallelism (TP)
Split each layer's weight matrices across GPUs. In a transformer MLP, the first matrix is split by columns and the second by rows, so each GPU computes its share and one all-reduce combines the result; attention splits by heads the same way.
- Cuts weight and activation memory per GPU by the TP degree.
- Communication: collective operations inside every layer, forward and backward. Only fast links (NVLink within a server) keep up, so TP is usually limited to the GPUs in one server, typically 8.
Pipeline parallelism (PP)
Put groups of consecutive layers on different GPUs (stages). Activations flow forward from stage to stage, gradients flow back.

- Communication is small: only the activations at stage boundaries, so it works across servers.
- The cost is the bubble: at the start of a step, later stages wait for the first micro-batch; at the end, earlier stages wait for the last gradients. Split the batch into m micro-batches across p stages and the idle fraction is about (p − 1) / (m + p − 1): 27% for 4 stages and 8 micro-batches, 10% for 8 stages and 64.
- Schedules matter: 1F1B (one forward, one backward, alternating) limits how many micro-batches' activations a stage must hold; interleaved schedules give each GPU several smaller stages to shrink the bubble further.
Combining them

Large runs use 3D parallelism: TP inside a server, PP across a few servers, DP across copies of that group. The rule is to put the most talkative split on the fastest link. Long-context training adds sequence or context parallelism (split the sequence across GPUs), and mixture-of-experts models add expert parallelism (different experts on different GPUs, tokens routed with all-to-all).
Trade-offs and costs
- DP: simple and efficient, but memory per GPU is unchanged and the global batch grows with GPU count, which may hurt convergence beyond some size.
- TP: fine-grained communication; efficiency drops quickly beyond one server.
- PP: bubbles, activation memory for in-flight micro-batches, load balancing (stages must have similar compute), and more complex code.
- Everything: more parallelism means more communication. Measured efficiency is usually reported as MFU (model FLOPs utilisation: useful FLOPs / peak FLOPs); large runs commonly land somewhere around 35–55%.
In production
- Frameworks do the plumbing: PyTorch FSDP and DTensor, DeepSpeed, Megatron-LM / Megatron Core, and higher-level trainers built on them. Your job is choosing degrees and checking MFU.
- A common recipe: start with FSDP (sharded DP) alone up to tens of billions of parameters; add TP within the node when per-GPU memory or batch-size limits bite; add PP when the model spans many nodes.
- Profile communication: if all-reduce or all-gather time isn't hidden behind compute, adjust bucket sizes, overlap settings, or the parallel layout.
When to reach for it
Any training or fine-tuning that doesn't fit on one GPU or takes too long on one. For fine-tuning models up to roughly 70B, sharded data parallel (FSDP/ZeRO-3), often with LoRA, is usually enough; full 3D layouts are for pretraining on hundreds or thousands of GPUs.
Example
Pretraining a 70B model on 512 GPUs (64 servers of 8). Training state at 16 bytes per parameter is 1.12 TB, too much for 8 GPUs even if perfectly split, before activations. Layout: TP = 8 within each server (each GPU holds 1/8 of every layer), PP = 4 across 4 servers (20 layers per stage), DP = 16 across 16 such groups, with ZeRO-1 sharding the optimizer state across the DP dimension. With 32 micro-batches per step the bubble is 3/35 ≈ 9%. Measured MFU is 42%; profiling shows DP gradient all-reduce is fully overlapped. (Example numbers.)
Common mistakes
- Using tensor parallelism across servers over the slower network.
- Too few micro-batches for the number of pipeline stages, so the bubble dominates.
- Unbalanced stages (the embedding or output layer making one stage slower).
- Assuming data parallelism saves memory; plain DP doesn't, sharded DP does.
- Ignoring global batch size when scaling DP: learning rate and convergence change.
Say this in the interview
"Data parallelism copies the model and splits the batch, with a gradient all-reduce each step; it scales throughput but not memory unless it's sharded like ZeRO or FSDP. Tensor parallelism splits each layer's matrices and talks in every layer, so it stays inside an NVLink server. Pipeline parallelism splits layers into stages, talks only at stage boundaries, and pays a bubble of about (p − 1)/(m + p − 1), so you use many micro-batches. Big runs combine them: TP inside the node, PP across nodes, DP across groups, matching the chattiest split to the fastest link, and you judge the layout by MFU."
Interview questions on this
- Why doesn't plain data parallelism let you train a bigger model? Every GPU still holds all weights, gradients and optimizer state.
- Why keep tensor parallelism within a node? It needs collectives in every layer; only NVLink-class bandwidth keeps that cheap.
- Compute the bubble for 8 stages and 16 micro-batches. 7 / 23 ≈ 30%.
- How does 1F1B differ from running all forwards then all backwards? It frees activations sooner, so each stage holds fewer micro-batches at once.
- What is MFU and what's a reasonable value? Useful FLOPs / peak; roughly 35–55% for large runs.
- How would you parallelise fine-tuning a 13B model on one 8-GPU server? FSDP across the 8 GPUs, maybe with activation checkpointing; no PP needed.
Used in
- AI Engineering: ZeRO and FSDP: Sharding Training State; Training Memory Math; GPU Interconnects and Collectives; Checkpoints and Fault Tolerance in Long Training Runs; Mixed Precision Training.
- AI System Design: Design a Fine-tuning Platform.