How Would You Cut LLM Latency in Half?
On this page
Problem statement
"Our RAG chat assistant takes about 3 seconds before the first word appears and up to 12 seconds to finish. Product wants it twice as fast. How would you do it?"
A favourite practical question because it separates people who know where LLM latency comes from from people who only know "use a smaller model".
Why it happens

- Serial steps before the model: an extra LLM call to rewrite the query, retrieval, reranking and guardrail checks, each waiting for the previous one.
- Long prompts: many retrieved chunks and a long system prompt make prefill slow, which raises TTFT.
- Queueing: at peak load, requests wait for GPU capacity.
- Long answers: total time ≈ TTFT + output tokens × TPOT; a 400-token answer at 25 ms per token is 10 s of generation (see Streaming and Latency).
- Model size: bigger models are slower per token.
Options
Weakswitch to a smaller model
It lowers TPOT and some prefill time, but leaves the serial pipeline untouched, and may hurt quality with no measurement to show it. If most of the time is outside the model, the gain is small.
Goodmeasure, stream, and trim the prompt
Add per-stage timing and look at p95. Stream tokens so users see text immediately. Retrieve fewer, better chunks (rerank to 5 instead of 20) and trim the system prompt; cap the answer length. These usually cut TTFT and total time noticeably with little risk.
Strongattack the critical path with three kinds of levers

- Do less: skip query rewriting for questions that don't need it (or use a small, fast model); cache the stable prompt prefix; semantic-cache frequent questions; return shorter, structured answers.
- Parallelise: run the input guardrail at the same time as retrieval; issue independent tool calls together; start retrieval while the user is still typing for predictable flows.
- Speed up the model: route easy questions to a smaller model; use quantized weights and speculative decoding where you self-host; add capacity or separate interactive from batch traffic to kill queueing.
- Guard quality: run the eval set after each change; keep a latency budget per stage.
How to measure it
- p50 and p95 of TTFT, TPOT and end-to-end time, overall and per stage, under realistic load.
- Output length distribution (tokens per answer).
- Cache hit rates (prefix, semantic) and queue time.
- Quality guardrails: answer correctness and faithfulness on the eval set must not drop.
Example answer flow
Using the example breakdown: the guardrail moves in parallel with retrieval (saves about 400 ms), query rewriting goes to a small model or is skipped for simple questions (about 700 → 150 ms), reranking keeps 5 of 20 candidates (300 → 120 ms), and the prompt shrinks from 18k to 6k tokens with a cached system prefix (prefill about 1,400 → 500 ms). New TTFT ≈ 150 + 150 + max(250 + 120, 400) + 500 ≈ 1.2 s, from 3.2 s. For total time, capping answers at about 200 tokens and routing easy questions to a faster model halves generation time. Then confirm on the eval set that correctness held. (All numbers are illustrative.)
Follow-up questions
- Which would you prioritise, TTFT or total time, for a chat UI? For an API used by another service?
- How does batching on the inference server affect your latency numbers?
- How do you stream an answer that must pass an output guardrail?
- When does speculative decoding help, and when doesn't it?
- Latency is fine at p50 but terrible at p95. What do you look at?
- How would you test that a latency fix didn't reduce answer quality?
Related
- Concepts: Streaming and Latency: TTFT and TPOT; Prefill, Decode and the KV Cache; Prefix Caching; Semantic Caching; Speculative Decoding; Continuous Batching.
- Practice: How Would You Cut LLM Cost by 10x?
- AI System Design: Design ChatGPT; Design an LLM Inference Platform.