Streaming and Latency: TTFT and TPOT
On this page
An assistant takes eight seconds to answer. Users call it "slow" and many leave before the answer appears. The team streams the reply instead: the first words show up in under half a second, the full answer still takes eight seconds, and complaints drop. Then a power user asks for a long report and it crawls out word by word. These are two different kinds of slowness, and LLM latency is always described with two numbers: time to first token (TTFT) and time per output token (TPOT).
How it works

The two phases of a request
- Before the first token: the request travels through your gateway and backend, may wait in a queue on the inference server, and then the model reads the whole prompt at once (prefill). This phase ends with the first output token: TTFT.
- Generation: the model produces one token per step (decode), each step reading the model's weights again. The gap between tokens is TPOT (also called inter-token latency); its inverse is tokens per second.

Total time ≈ TTFT + (output tokens × TPOT). For example, with TTFT of 0.5 s, TPOT of 25 ms (40 tokens per second) and a 400-token answer: 0.5 + 400 × 0.025 = 10.5 s. Streaming doesn't change that number, but the user sees text after 0.5 s.
Streaming to the client
Instead of waiting for the full answer, the server sends tokens as they are produced, usually over Server-Sent Events (SSE) (a long-lived HTTP response that sends small data: events) or WebSockets for two-way apps such as voice. Every hop (API gateway, load balancer, your backend) must pass chunks through instead of buffering the whole response, and timeouts must allow long-lived connections.
What moves each number

- TTFT grows with prompt length (prefill work), queueing when the server is busy, model size, network hops, and anything you do before calling the model (retrieval, tool calls, guardrail checks).
- TPOT grows with model size and the number of requests batched together on the GPU, and shrinks with faster memory bandwidth, quantization and speculative decoding (see AI Infra · Inference).
Trade-offs and costs
- Throughput vs latency. Servers batch many requests together to use the GPU well. Bigger batches raise total tokens per second (cheaper) but make each user's TPOT worse.
- Guardrails vs streaming. If you must check the output before showing it, you can't stream it raw. Options: check in chunks (sentence by sentence), stream while running checks and retract on failure, or check only the input.
- Reasoning models spend time "thinking" before the first visible token, so their TTFT is high by design; show progress to users.
- Prompt length. Retrieval that adds 20,000 tokens of context raises TTFT noticeably; retrieve less, better.
In production
- Measure both, by percentile. Track p50 and p95 TTFT and TPOT, plus end-to-end time, per model and per route. Averages hide queueing spikes.
- Set budgets. A chat UI might target TTFT under about 1 s at p95; a voice assistant needs much less. Write the budget down per feature (these targets are examples, not standards).
- Watch the pipeline before the model. Retrieval, reranking and tool calls often take more time than the model's TTFT.
- Failure modes: a proxy buffering the stream so users see nothing until the end; idle-timeout cutting long streams; clients that can't handle partial JSON; queueing under load making TTFT jump from 0.5 s to 10 s.
When to reach for it
Any user-facing LLM feature: chat, copilots, voice, search answers. For batch jobs (overnight summarisation), throughput and cost matter more than TTFT.
Example
A support chat has p95 TTFT of 3.2 s. Breaking it down: 150 ms gateway and auth, 900 ms retrieval plus reranking, 400 ms input guardrail, 1.8 s model TTFT (an 18,000-token prompt). Fixes: run the guardrail in parallel with retrieval (−400 ms), retrieve 6 chunks instead of 20 (prompt down to about 6,000 tokens, model TTFT around 0.7 s), and cache the static system prompt prefix. New p95 TTFT is about 1.5 s, with no change of model.
Common mistakes
- Reporting one "latency" number without separating TTFT and TPOT.
- Optimising the model while the pipeline is slow (retrieval, tools, guardrails).
- Buffering proxies that silently disable streaming.
- Ignoring output length: asking for shorter answers is often the cheapest speed-up.
- Measuring averages instead of p95 under real load.
Say this in the interview
"LLM latency has two parts: TTFT, which covers queueing and prefill and grows with prompt length, and TPOT, the time per generated token, set by model size, batch load and memory bandwidth. Total time is roughly TTFT plus output tokens times TPOT. I'd stream over SSE so users read early, measure p95 of both, cut prompt length and pre-model steps for TTFT, and use a smaller or quantized model, speculative decoding or shorter outputs for TPOT."
Interview questions on this
- Why does a longer prompt increase TTFT but not TPOT much? Prefill processes every prompt token before the first output; each decode step only adds one token (though attention over a longer context costs a little more).
- How does batching affect latency? It improves throughput but each request shares the GPU, so TPOT rises.
- How would you stream a response through an API gateway? Use SSE or WebSockets, disable response buffering, and raise idle timeouts.
- How do you combine streaming with output guardrails? Check in chunks, or stream and retract on violation, or rely on input checks plus post-hoc review for low-risk content.
- A user says it's slow; what do you measure first? End-to-end breakdown: network, pre-model steps, queue, TTFT, TPOT and output length.
Used in
- AI Engineering: Prefill, Decode and the KV Cache; Continuous Batching; Speculative Decoding; How Would You Cut LLM Latency in Half?; Serving LLMs: Metrics and Autoscaling.
- AI System Design: Streaming and latency (TTFT/TPOT); Design ChatGPT; Design ChatGPT Voice.