Compute-Bound vs Memory-Bound: The Roofline View
On this page
Two engineers profile the same LLM server. One says "the GPU is at 3% of its peak FLOPS, the kernels must be bad." The other says "we're at 90% of memory bandwidth, the kernels are fine." Both are looking at decode at batch size 1, and the second one is right. The roofline model is a one-line way to tell whether an operation is limited by math or by memory, and therefore which optimisation can possibly help. It's the single most useful mental model in AI infra interviews.
How it works
Arithmetic intensity
For any operation, count:
- FLOPs: floating-point operations it performs.
- Bytes: data it must read from and write to GPU memory (HBM).
Arithmetic intensity = FLOPs / bytes. A GPU can do P FLOPs per second and move B bytes per second, so the best speed an operation can reach is:
attainable FLOPS = min(P, intensity × B)

Plotted on log axes, that's a sloped line (memory roof) meeting a flat line (compute roof). Where they meet is the ridge point, P / B. For an H100 SXM-class GPU (about 989 dense BF16 TFLOPS, 3.35 TB/s) the ridge is about 295 FLOPs per byte. Below it, an operation is memory-bound; above it, compute-bound.
Where LLM operations land
Numbers from the included roofline.py, for an 8,192 × 8,192 linear layer in BF16:
- Decode, batch 1: each 2-byte weight is used for one multiply-add (2 FLOPs), so intensity ≈ 1. The GPU can reach only about 3 TFLOPS: 0.3% of peak, and that's the hardware limit, not bad code.
- Decode, batch b: each weight is reused b times, so intensity ≈ b. Batch 64 reaches about 211 TFLOPS.
- Prefill or training with 4,096 tokens: intensity ≈ 2,048, well above the ridge: compute-bound.
- Elementwise ops (adds, activations, unfused norms): about 0.3 FLOPs per byte, firmly memory-bound.
- Decode attention over a KV cache: each cached key and value is used once per query head in its group, intensity around 8 with grouped-query attention: memory-bound regardless of batch, since each request has its own cache.
Batch is free until the ridge

Because decode is memory-bound, the step time is the time to read the weights: about 40 µs for this layer whether the batch is 1, 8, 64 or 256. Throughput grows almost linearly with batch until the ridge, after which time per step grows with the batch. That one fact explains continuous batching, why serving cost per token falls with load, and why KV cache memory (which caps batch size) is so valuable.
Trade-offs and costs

- Memory-bound: raise intensity (bigger batches, kernel fusion so intermediates stay in SRAM, FlashAttention) or cut bytes (quantized weights and KV cache; speculative decoding verifies several tokens per weight read). More FLOPS won't help.
- Compute-bound: do less math (mixture of experts, smaller models), use faster math (FP8 tensor cores), or better kernels. More bandwidth won't help.
- Below both roofs: the hardware isn't the limit at all: kernel launch overhead, Python, CPU data loading, synchronisation. Profile before optimising anything.
- The model is simplified. Real kernels have caches, overlap and imperfect peak numbers; treat the roofline as a bound and a direction, not a prediction.
In production
- When an LLM service is "slow", first ask which phase: prefill is compute-bound, decode is memory-bound, and their fixes differ.
- When comparing GPUs for inference, compare memory bandwidth and capacity per dollar; for training, compare usable FLOPS at your precision and interconnect.
- Profilers (Nsight Compute) report each kernel's achieved FLOPS and bandwidth and place it on a roofline directly.
- MFU (model FLOPs utilisation) is the training-side summary; for decode, a better summary is achieved memory bandwidth utilisation.
When to reach for it
Any performance question: "why is decode slow?", "will FP8 help?", "should we buy GPU X?", "why does batching help so much?", "is this kernel good?". It also lets you sanity-check claims quickly in interviews.
Example
A team serves a 70B model with 4-way tensor parallelism and wants lower TPOT. Per GPU, a decode step reads about 35 GB of weights (140 GB / 4); at 3.35 TB/s that's about 10.4 ms, so TPOT can't go below roughly 10 ms however good the kernels are. Options that move the memory roof: FP8 weights (about 5.2 ms floor), 8-way tensor parallel (also about 5.2 ms, but more communication), or speculative decoding (several tokens per weight read). Buying a GPU with more FLOPS but the same bandwidth would change nothing. (Example numbers.)
Common mistakes
- Judging decode efficiency by % of peak FLOPS.
- Expecting FP8 compute to speed up a memory-bound op; the gain there comes from reading fewer bytes.
- Forgetting KV cache reads in decode, which grow with context length and don't benefit from batching.
- Optimising kernels when the time is spent in overhead (small batches, Python, data loading).
Say this in the interview
"Arithmetic intensity is FLOPs per byte moved. Attainable speed is the minimum of peak FLOPS and intensity times bandwidth; on an H100-class GPU the ridge is around 300 FLOPs per byte. Decode at batch 1 has intensity about 1, so it's memory-bound and runs at a fraction of a percent of peak by design; intensity grows with batch size, which is why batching makes decode cheap until you hit the ridge. Prefill and training are compute-bound. So for decode I'd cut bytes or raise reuse (batching, quantization, fusion, speculative decoding); for prefill I'd use faster math."
Interview questions on this
- Compute the ridge point for a GPU with 400 TFLOPS and 2 TB/s. 200 FLOPs per byte.
- What is the arithmetic intensity of decode at batch size 32? About 32 for the weight matrices.
- Will doubling FLOPS speed up decode? Not unless the batch is past the ridge; bandwidth is the limit.
- Why is an unfused layer norm slow? Very low intensity; it reads and writes the whole tensor for a few FLOPs per element.
- What's the minimum TPOT for an 8B BF16 model on a 3.35 TB/s GPU? About 16 GB / 3.35 TB/s ≈ 4.8 ms.
- Why doesn't batching help decode attention as much as the linear layers? Each request reads its own KV cache, so there's no reuse across the batch.
Used in
- AI Engineering: GPU Basics for AI Engineers; Prefill, Decode and the KV Cache; Continuous Batching; FlashAttention; Quantization: INT8, INT4 and FP8; Speculative Decoding; GPU Utilization and Cost.
- AI System Design: Design an Inference Platform.