Speculative Decoding

Updated Nov 16, 2026 · 5 min read
On this page

    A coding assistant streams at 35 tokens per second on a 70B model, and users say it feels sluggish on long answers. The GPU isn't out of compute; each decode step is waiting on memory, reading 140 GB of weights to produce one token per request. A 1B model on the same GPU runs ten times faster but writes worse code. Speculative decoding gets most of the small model's speed with exactly the big model's output: the small model guesses several tokens ahead, and the big model checks all the guesses in one pass.

    How it works

    The draft model proposes four tokens; the target checks them in one pass, keeps the three that match, and supplies its own fourth token.
    The draft model proposes four tokens; the target checks them in one pass, keeps the three that match, and supplies its own fourth token.

    Draft, verify, accept

    1. Draft. A cheap draft model generates k tokens (say 4) one after another.
    2. Verify. The big target model runs one forward pass over the context plus all k draft tokens. Like prefill, that pass computes the target's prediction at every position at once.
    3. Accept. Walk through the drafts in order. Keep each one the target agrees with; at the first disagreement, throw away the rest and use the target's own token there. If all k pass, the target's prediction after the last draft is a free extra token.

    Every round produces between 1 and k + 1 tokens for one target pass. The trick works because decode is memory-bound: checking 5 positions costs about the same as generating 1, since the weights are read once either way.

    Why the output is unchanged

    Accept a draft token with probability min(1, p/q); on rejection, sample from the part of p that q under-covers.
    Accept a draft token with probability min(1, p/q); on rejection, sample from the part of p that q under-covers.

    With greedy decoding, "agrees" just means the draft token equals the target's top token, so the output is identical to running the target alone. With sampling (temperature > 0), the rule is:

    • Accept draft token x with probability min(1, p(x) / q(x)), where p is the target's distribution and q the draft's.
    • If rejected, sample a replacement from max(p − q, 0), renormalised.

    The combined result follows p exactly, not approximately. The included spec_demo.py checks this on a toy distribution: after 200,000 samples the empirical frequencies match the target to three decimals, and the acceptance rate matches the theory, α = Σ min(p, q) = 0.79.

    How much faster

    Expected tokens per target pass for acceptance rates 0.6, 0.8 and 0.9 as the draft length grows.
    Expected tokens per target pass for acceptance rates 0.6, 0.8 and 0.9 as the draft length grows.

    If each draft token is accepted with probability α, the expected tokens per target pass is (1 − α^(k+1)) / (1 − α). With α = 0.8 and k = 4 that is 3.4 tokens per pass. Real speedup is lower, because the draft model's own time and the slightly bigger verify pass both cost something; 2–3× on decode is a typical published range.

    Where drafts come from

    • A small model from the same family (same tokenizer), for example 1B drafting for 70B.
    • Extra prediction heads on the target itself (Medusa) or a light draft head that reads the target's hidden states (EAGLE). No second model to deploy, and acceptance is often higher.
    • Prompt lookup / n-gram drafting: copy upcoming tokens from the prompt. Very effective for editing code or summarising a document, where the output repeats the input.

    Trade-offs and costs

    • It helps latency, not always throughput. At large batch sizes the GPU is closer to compute-bound, and verify work on rejected tokens is wasted. Many servers turn speculation down or off as load rises.
    • Acceptance is workload-dependent. Code and structured text accept well; creative, high-temperature text accepts poorly. Measure α on your own traffic.
    • Memory. A draft model needs its own weights and KV cache on the GPU.
    • Draft length. Longer drafts help only if α is high; the curve flattens and each rejected draft is wasted work.
    • Complexity. Rolling back the KV cache for rejected tokens and keeping draft and target caches aligned is engine work.

    In production

    • vLLM, SGLang and TensorRT-LLM support draft models, EAGLE/Medusa-style heads and n-gram drafting through configuration.
    • Track acceptance rate, mean accepted length, TPOT and throughput, per route. A drop in acceptance after a model upgrade often means the draft no longer matches the target.
    • Load test at your real batch sizes; a 2.5× win at batch 1 can shrink to nothing at batch 64.
    • Some hosted APIs offer it as "predicted outputs": you pass text you expect the answer to contain (the file being edited) and it is used as the draft.

    When to reach for it

    Latency-sensitive decode with spare compute: chat and coding assistants at modest load, code editing where the output mostly repeats the input, long answers where TPOT dominates. Not the first tool for offline batch jobs that already run the GPU at full compute.

    Example

    A code-editing feature rewrites files of about 800 tokens with small changes. With prompt-lookup drafting (k = 8, drafts copied from the original file), mean accepted length is 5.6 tokens per target pass and TPOT drops from 28 ms to 9 ms per token. For the general chat route, a 1B draft model gives α ≈ 0.65 and only a 1.4× gain at peak batch sizes, so the team keeps it for the off-peak, low-batch tier only. (Example numbers.)

    Common mistakes

    • Claiming it changes output quality. With the proper acceptance rule it doesn't; the output distribution is the target's.
    • Using a draft with a different tokenizer without handling the mismatch.
    • Measuring only at batch size 1.
    • Choosing k without measuring α.
    • Forgetting the draft model's memory, which takes space from the KV cache.

    Say this in the interview

    "Decode is memory-bound, so one target pass can check several tokens almost as cheaply as it generates one. A small draft model, extra heads or n-gram lookup propose k tokens; the target verifies them in one pass and keeps the prefix it agrees with, plus one token of its own. With the acceptance rule min(1, p/q) and resampling from the leftover, the output distribution is exactly the target's. Expected tokens per pass is (1 − α^(k+1))/(1 − α), so it pays when acceptance is high, like code editing. It helps latency at low and medium load; at high batch sizes I'd measure and maybe turn it off."

    Interview questions on this

    1. Why can the target check k tokens for about the cost of one? Decode is memory-bound; verifying k positions reads the weights once, like prefill.
    2. Does speculative decoding change the output? No: greedy output is identical and sampled output has the same distribution.
    3. What decides the speedup? Acceptance rate, draft length, draft cost, and batch size.
    4. Why does it help less at high load? The GPU becomes compute-bound and rejected tokens waste compute.
    5. What are the options for a draft? A small model with the same tokenizer, Medusa/EAGLE heads, or n-gram prompt lookup.
    6. How would you choose k? Measure acceptance on real traffic and pick the k that maximises tokens per second, not tokens per pass.

    Used in

    • AI Engineering: Prefill, Decode and the KV Cache; Streaming and Latency: TTFT and TPOT; How Would You Cut LLM Latency in Half?; Compute-Bound vs Memory-Bound: The Roofline View; Distillation: Making a Small Model Act Big.
    • AI System Design: Design Cursor; Design an Inference Platform.