GPU Interconnects and Collectives: NVLink, InfiniBand, All-Reduce
On this page
A team scales a training job from one 8-GPU server to eight servers and expects close to 8× the throughput. They get 3.5×. The GPUs are identical; what changed is how they talk. Inside a server, GPUs exchange data over NVLink at hundreds of GB/s; between servers, over a network roughly 18× slower per GPU. Every step of distributed training and multi-GPU inference ends with GPUs combining data through collective operations, and how fast those run depends on the interconnect, the algorithm, and whether the work can hide behind compute.
How it works
Two networks

- Scale-up (inside a server): NVLink and NVSwitch. On an H100-class server, every GPU reaches every other at about 900 GB/s of total NVLink bandwidth (around 450 GB/s each way). Newer rack-scale systems extend this NVLink domain to dozens of GPUs.
- Scale-out (between servers): InfiniBand or RoCE (RDMA over Converged Ethernet). Training clusters typically give each GPU its own 400 Gb/s network card, about 50 GB/s, with RDMA so data moves GPU-to-GPU without going through the CPU (GPUDirect RDMA). Switches are arranged leaf-spine (fat tree), ideally non-blocking, often "rail-optimised" so GPU k in every server shares a leaf switch.
- PCIe links the GPUs to the CPU and, in cheaper servers without NVLink, to each other, at around 64 GB/s.
The collectives

- All-reduce: every GPU ends with the sum of everyone's data. Data-parallel gradient sync, and tensor-parallel layer outputs.
- Reduce-scatter: every GPU ends with 1/n of the sum. ZeRO/FSDP gradients.
- All-gather: every GPU ends with every piece. FSDP weights before compute, sequence-parallel activations.
- Broadcast: one GPU's data to all. Initial weights.
- All-to-all: each GPU sends a different piece to each other GPU. Mixture-of-experts token routing; the hardest pattern for the network.
Libraries such as NCCL (NVIDIA) and RCCL (AMD) implement them and pick algorithms based on topology.
Ring all-reduce and its cost

Split the data into n chunks. In the reduce-scatter phase, each GPU passes a chunk to its neighbour, which adds its own copy and passes it on; after n−1 steps each GPU holds the full sum of one chunk. In the all-gather phase, the finished chunks travel around the ring for another n−1 steps. Each GPU sends about 2(n−1)/n × the data size, nearly 2× regardless of n, so bandwidth cost doesn't grow with GPU count. The included ring_allreduce.py simulates it and checks every GPU ends with the exact sum.
Time ≈ 2(n−1)/n × size / bandwidth + 2(n−1) × per-step latency. For 14 GB of gradients (a 7B model):
- 8 GPUs over NVLink: about 55 ms.
- 64 GPUs over 400 Gb/s: about 550 ms, set by the slowest link in the ring.
- 512 GPUs: similar bandwidth time, but the latency term grows with n, which is why NCCL uses tree and hierarchical algorithms at scale (reduce within the server over NVLink first, then across servers).
Trade-offs and costs
- Overlap or wait. Communication that runs while the GPU computes the next layer is nearly free; communication that blocks the step is pure loss. Gradient bucketing, FSDP prefetching and pipelined schedules all exist to overlap.
- Bandwidth vs latency. Big messages are limited by bandwidth; many small messages by latency. Fuse small tensors into buckets.
- Place traffic on the right link. Tensor parallelism inside the NVLink domain; pipeline and data parallelism across the network (see Parallelism for Training).
- The network is shared and imperfect: congestion from other jobs, a flapping link, or one slow GPU slows every collective that includes it.
- Cost: a non-blocking InfiniBand fabric is a large share of a training cluster's price; inference-only clusters often use cheaper Ethernet.
In production
- Benchmark the fabric before training with
nccl-tests(all_reduce_perf) and compare measured "bus bandwidth" with the hardware's numbers; repeat after any hardware change. - Turn on NCCL debug logging when a job hangs; set collective timeouts so a stuck rank fails fast (see Checkpoints and Fault Tolerance).
- Watch per-step communication time in the profiler; if it isn't overlapped, adjust bucket sizes, prefetching or the parallel layout.
- Use topology-aware scheduling so a job's GPUs share leaf switches (see GPU Job Scheduling and Queues).
When to reach for it
Whenever a job spans more than one GPU: choosing a parallel layout, debugging poor scaling, designing a training cluster, or picking hardware for tensor-parallel inference of large models.
Example
The team from the opening profiles their 64-GPU run of a 7B model with plain data parallelism. Compute per step is 450 ms; the 14 GB gradient all-reduce takes about 550 ms over the network and only half of it overlaps with backward, so a step takes about 725 ms instead of 450. Fixes: switch to hybrid sharding (FSDP inside each server over NVLink, replication across servers), increase the micro-batch so compute per step doubles, and tune bucket sizes for overlap. Step time drops to about 940 ms for twice the work: an effective 1.5× gain, and scaling efficiency rises from about 62% to about 96%. (Example numbers.)
Common mistakes
- Putting tensor parallelism across servers.
- Assuming all-reduce cost grows with GPU count. Ring bandwidth cost is nearly constant per GPU; latency and stragglers are what grow.
- Not testing the fabric, then blaming the model code for slow scaling.
- Ignoring all-to-all when moving to mixture-of-experts models.
- No timeouts, so one bad rank hangs the whole job silently.
Say this in the interview
"There are two networks: NVLink inside a server, about 900 GB/s per GPU, and InfiniBand or RoCE between servers, about 50 GB/s per GPU with RDMA. Training uses collectives (all-reduce for data-parallel gradients and tensor-parallel outputs, reduce-scatter and all-gather for FSDP, all-to-all for MoE), implemented by NCCL. A ring all-reduce sends about twice the data per GPU regardless of GPU count, so its time is roughly 2 × size / slowest link bandwidth plus a latency term that grows with n. I'd keep the chattiest traffic on NVLink, overlap communication with compute, and benchmark the fabric with nccl-tests."
Interview questions on this
- How long does a ring all-reduce of 10 GB take on 16 GPUs at 50 GB/s? About 2 × 15/16 × 10 / 50 ≈ 0.375 s plus latency.
- What's the difference between all-reduce and reduce-scatter? All-reduce gives everyone the full sum; reduce-scatter gives each GPU only its share.
- Why is MoE training hard on networks? All-to-all token routing is bursty and hits every link.
- What is GPUDirect RDMA? The network card reads and writes GPU memory directly, bypassing the CPU.
- Why use hierarchical all-reduce? Reduce inside the fast NVLink domain first, then send less data over the slower network, with fewer latency steps.
- Training slowed by 30% overnight with no code change. What do you check? A degraded link or GPU, a straggler node, network congestion, nccl-tests bandwidth.
Used in
- AI Engineering: Parallelism for Training; ZeRO and FSDP; GPU Basics for AI Engineers; GPU Job Scheduling and Queues; Checkpoints and Fault Tolerance in Long Training Runs; Serving LLMs: Metrics, Autoscaling and Disaggregated Prefill.
- AI System Design: Design a Fine-tuning Platform; Design an Inference Platform.