Prefix Caching
On this page
A support assistant sends every request with the same 3,000-token prefix: system prompt, tool definitions and refund policy. The user's actual question adds about 200 tokens. So 94% of the prefill work on every call recomputes keys and values the server computed for the previous caller a second ago. Agents are worse: each step resends the whole growing transcript. Prefix caching keeps the KV cache of prompt prefixes the server has already seen and reuses it whenever a new request starts with exactly the same tokens.
How it works

Reusing KV blocks
The keys and values for a token depend only on that token and the tokens before it. So if two prompts share their first 3,000 tokens, the KV cache for those 3,000 tokens is identical, and the second request can skip prefilling them.
With a paged KV cache (see PagedAttention), this is cheap to implement:

- Split the prompt into blocks (for example 16 tokens).
- Give each full block a key: hash(previous block's key, this block's tokens). Chaining means a block only matches if every token before it matches too.
- Look the keys up in a table of cached blocks, in order, and stop at the first miss.
- Point the request's block table at the hit blocks (reference count + 1), prefill only the remaining tokens, and register the new blocks so later requests can hit them.
- When memory is needed, evict cached blocks no running request uses, least recently used first.
vLLM calls this automatic prefix caching; SGLang's RadixAttention keeps cached prefixes in a radix tree, which suits branching agent and multi-turn traffic.
What it saves
- Prefill compute: the 3,200-token request prefills about 200 tokens.
- TTFT: prefill time shrinks in proportion, often from hundreds of milliseconds to tens.
- Memory: many running requests share one copy of the prefix blocks, so more requests fit in a batch.
It doesn't speed up decode: generated tokens are still produced one at a time.
Provider prompt caching
Hosted APIs offer the same thing as a billing feature. Some cache long prefixes automatically, others let you mark cache breakpoints in the request. Cached input tokens are billed at a fraction of the normal input price and come back faster; entries expire after minutes of disuse. Check your provider's current docs for the minimum length, time-to-live and discount.
Trade-offs and costs
- Exact match only. One different token early in the prompt (a timestamp, a user ID, a reordered tool list) makes everything after it a miss. Semantic caching is the different technique that matches meaning; it returns a stored answer and skips the model entirely.
- Memory competition. Cached prefixes live in the same pool as active requests and get evicted under load, so hit rates drop exactly at peak.
- Routing matters. With many replicas, a prefix is cached on only the replicas that saw it. Without cache-aware routing (sending requests with the same prefix to the same replica), the hit rate falls as you scale out.
- Multi-tenant isolation. Sharing cached blocks across tenants can leak information through timing (a hit is measurably faster). Scope the cache key per tenant where that matters.
In production

- Design prompts for it: system prompt and tool definitions first, then long shared documents, then conversation history, then the new message. Keep dates and request IDs out of the top. Serialise tool schemas and JSON deterministically.
- Agents and chat: each turn appends to the previous prompt, so the whole history before the new message is a hit, as long as you don't rewrite or re-summarise the start of it every turn.
- Route by prefix: hash the first N tokens (or the session ID) at the load balancer and keep similar requests on the same replica, with a fallback when that replica is overloaded.
- Measure: prefix cache hit rate (tokens served from cache / prompt tokens), TTFT for hits vs misses, cached-token share on the provider bill.
When to reach for it
Long shared system prompts, few-shot examples, tool-heavy agents, multi-turn chat, many questions over the same document, and evaluation runs that send the same instructions thousands of times. It matters little when prompts are short or unique.
Example
The support assistant runs on 4 replicas with round-robin routing. Prefix hit rate is 31%, and p50 TTFT is 420 ms. Two changes: the "current time" line moves from the top of the system prompt to just before the user message, and the load balancer routes by tenant ID with consistent hashing. Hit rate rises to 88% of prompt tokens, p50 TTFT drops to 110 ms, and prefill GPU time falls by about three quarters, so the same replicas absorb 40% more traffic. (Example numbers.)
Common mistakes
- Putting volatile content first: timestamps, user names or retrieved documents before the fixed instructions.
- Rebuilding prompts non-deterministically: tool lists in random order, JSON with unsorted keys, whitespace changes.
- Confusing it with semantic caching, which reuses whole answers for similar questions.
- Ignoring routing when scaling from one replica to many.
- Assuming it speeds up long outputs: it only shortens prefill and TTFT.
Say this in the interview
"Keys and values for a token depend only on the tokens before it, so identical prompt prefixes have identical KV caches. With a paged cache, the server hashes each block chained with its predecessor, reuses the longest matching run of blocks, and prefills only the rest. It cuts prefill compute and TTFT and lets requests share memory. To get hits, put stable content first and dynamic content last, keep serialisation deterministic, and route requests with the same prefix to the same replica. Hosted APIs expose the same idea as discounted cached input tokens."
Interview questions on this
- Why can a cached prefix be reused but not a cached suffix? A token's keys and values depend on everything before it; the same text after a different prefix has different values.
- Your hit rate dropped from 80% to 20% after a deploy. What do you check? A changed prompt template, a new dynamic field near the top, tool order, and routing changes.
- How does prefix caching interact with load balancing? Hits only happen on replicas that hold the prefix; use prefix- or session-affinity routing.
- Prefix caching vs semantic caching? Exact token match that skips prefill vs similar-meaning match that skips the model call.
- What does it do to TTFT and TPOT? Lowers TTFT on hits; TPOT is mostly unchanged.
- Any security concern? Cross-tenant timing side channels; scope cache keys per tenant when needed.
Used in
- AI Engineering: Prefill, Decode and the KV Cache; PagedAttention and KV Cache Memory Management; Semantic Caching for LLM Apps; Context Windows and Token Cost; How Would You Cut LLM Cost by 10x?
- AI System Design: Design ChatGPT; Design Cursor; Design a Customer Support Agent.