Prefill, Decode and the KV Cache

Updated Nov 16, 2026 · 6 min read
On this page

    A team serves an 8B model on one 80 GB GPU. A 3,000-token prompt gets its first word back in about 150 ms, yet the answer then trickles out at a steady pace no matter how short the prompt was. When traffic doubles, nothing looks busy on the GPU's compute graphs, but new requests start waiting anyway. All three facts come from the same place: an LLM request runs in two phases, prefill and decode, and the KV cache that connects them decides how many users fit on a GPU.

    How it works

    Prefill reads the whole prompt in one parallel pass; decode then produces one token per step.
    Prefill reads the whole prompt in one parallel pass; decode then produces one token per step.

    Prefill: read the prompt in one pass

    The model runs every prompt token through every layer at once. That is one large matrix multiplication per weight matrix, with thousands of rows, so the GPU's math units stay busy: prefill is compute-bound. Its cost grows with prompt length, and it ends when the first output token is produced, so it sets most of time to first token (TTFT).

    Decode: one token per step

    To generate, the model appends the token it just picked and runs again. Each step handles one new token per request, yet it still has to read every weight from GPU memory. A forward pass does about 2 floating-point operations per parameter per token, while reading the weights moves 2 bytes per parameter in FP16, so decode does roughly one operation per byte. GPUs can do hundreds of operations per byte they read, so decode is memory-bound: the time per token is set by memory bandwidth, not by math. This sets time per output token (TPOT).

    A rough floor: an 8B model in FP16 is 16 GB of weights. A GPU with 3.35 TB/s of memory bandwidth (H100 SXM class) needs about 16 / 3,350 s ≈ 4.8 ms just to read them once, so one request alone can't go faster than about 200 tokens per second, however idle the math units are.

    The KV cache

    Attention lets each new token look at every earlier token through their keys (K) and values (V). Those K and V vectors don't change once computed, because a token can only see tokens before it. So the server keeps them in GPU memory, per layer and per request, and each decode step computes K and V for the new token only and appends them.

    Without a cache, each step recomputes every earlier token; with it, each step adds one token's keys and values.
    Without a cache, each step recomputes every earlier token; with it, each step adds one token's keys and values.

    Without the cache, generating 500 tokens after a 1,000-token prompt means about 625,000 token passes in total, because each step redoes the whole history. With it, about 1,500. Prefill is the step that fills the cache for the prompt; decode reads it and grows it by one entry per step.

    How big it gets

    KV cache size per token and what fits next to an 8B model on one 80 GB GPU (example numbers).
    KV cache size per token and what fits next to an 8B model on one 80 GB GPU (example numbers).

    Bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per value.

    • An 8B model shaped like Llama 3 8B (32 layers, 8 KV heads, head dim 128, FP16): 2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KB per token.
    • A 70B model shaped like Llama 3 70B (80 layers, 8 KV heads, head dim 128): ≈ 320 KB per token, so one 4,096-token conversation holds about 1.3 GB.

    On an 80 GB GPU with 16 GB of weights and some runtime overhead, about 50 GB is left for the cache: roughly 380,000 tokens, or about 90 conversations of 4,000 tokens at the same time. That number, not compute, is usually what caps the batch size, and the batch size is what sets cost per token.

    Trade-offs and costs

    • Batch size vs TPOT. Decode reads the weights once per step for the whole batch, so batching 64 requests costs little more per step than batching 1. Throughput rises almost linearly until the math units fill up or the KV cache runs out; each user's TPOT gets a little worse as the batch grows.
    • Long contexts eat batch slots. A 32,000-token request uses as much cache as eight 4,000-token ones.
    • Prefill interrupts decode. A long prompt arriving mid-batch takes over the GPU for its prefill and stalls every other user's next token. Servers split prefill into chunks or run it on separate machines (see Serving LLMs).
    • Shrinking the cache costs something. Grouped-query attention (fewer KV heads, used in most current open models), an FP8 or INT8 cache, and sliding-window attention all cut memory at some cost in quality or flexibility.

    In production

    • Serving engines such as vLLM, SGLang and TensorRT-LLM manage the cache for you: they allocate it in blocks (PagedAttention), reuse shared prefixes (prefix caching), and preempt requests when it fills.
    • Watch KV cache usage %, the number of waiting requests, and preemptions. A full cache with a growing queue means you need more GPUs or shorter contexts, even when compute utilisation looks low.
    • Measure TTFT and TPOT separately. TTFT problems point to prefill and queueing; TPOT problems point to batch size, model size and memory bandwidth.

    When to reach for it

    Any time you size, tune or debug LLM serving: choosing a GPU, setting max context length and max batch size, explaining why latency changes under load, or estimating cost per token. Application engineers need it too: long system prompts and big retrieved contexts cost prefill time and cache memory on every call.

    Example

    A chat feature serves an 8B model on one 80 GB GPU. Average conversations use 6,000 tokens (system prompt, history and answer), so each holds about 6,000 × 128 KB ≈ 0.77 GB of cache. 50 GB of cache space fits about 63 conversations at a time. Product wants a 32,000-token context option: those requests take about 4 GB each, so 10 of them would use most of the space. The team caps long-context requests at 10% of slots and routes them to a separate pool, and turns on prefix caching for the 2,000-token system prompt every request shares.

    Common mistakes

    • Treating LLM inference as compute-bound. Decode is limited by memory bandwidth and KV cache space; more FLOPs don't help a batch of one.
    • Forgetting the cache when sizing GPUs. Weights fit, but the cache for real context lengths doesn't, so the server runs with a tiny batch.
    • Counting model heads instead of KV heads. With grouped-query attention, a model with 64 query heads may have only 8 KV heads, an 8× smaller cache.
    • One latency number. TTFT and TPOT have different causes and different fixes.

    Say this in the interview

    "Each request has two phases. Prefill runs the whole prompt in one parallel pass; it is compute-bound and sets TTFT. Decode makes one token per step and must read all the weights each step, so it's memory-bandwidth-bound and sets TPOT. The KV cache stores each token's keys and values so decode only computes the new token. Its size is 2 × layers × KV heads × head dim × bytes per token, about 128 KB for an 8B model, and that memory, not compute, usually limits how many requests we can batch, which sets cost per token."

    Interview questions on this

    1. Why is decode memory-bound but prefill compute-bound? Prefill multiplies the weights by many tokens at once, so each byte read is used many times; decode uses each weight for one token per request per step.
    2. Compute the KV cache for a model with 80 layers, 8 KV heads, head dim 128, FP16, at 8K context. 320 KB per token × 8,192 ≈ 2.7 GB per sequence.
    3. Why does batching make decode cheaper? The weight reads are shared by every request in the batch, so throughput grows with batch size until compute or cache memory runs out.
    4. What limits batch size on a serving GPU? Usually KV cache memory, set by context length; sometimes compute for very large batches.
    5. How would you fit more concurrent users without new GPUs? Grouped-query attention models, an FP8 cache, quantized weights to free memory, prefix caching, shorter prompts, and paged allocation to cut waste.
    6. Why does a long prompt from one user slow down everyone else's tokens? Its prefill occupies the GPU between decode steps; chunked prefill or separate prefill machines fix it.

    Used in

    • AI Engineering: Streaming and Latency: TTFT and TPOT; Implement a KV Cache for Generation; Continuous Batching; PagedAttention and KV Cache Memory Management; Prefix Caching; Compute-Bound vs Memory-Bound: The Roofline View; How Many GPUs Do You Need to Serve a 70B Model?
    • AI System Design: Design ChatGPT; Design an Inference Platform.