How an LLM Picks the Next Token: Temperature, Top-k, Top-p
On this page
You build a feature that extracts invoice fields into JSON. In testing it is perfect; in production, one response in fifty has a made-up field name. Meanwhile the marketing team says the slogan generator repeats the same three phrases. Same model, opposite complaints, and the cause of both is one setting most people never look at: how the next token is chosen.
How it works
The model gives a distribution, not an answer
On every step, the model outputs a score (a logit) for every token in its vocabulary. Softmax turns the scores into probabilities that add up to 1. The model's job ends there. Decoding is the separate rule that turns those probabilities into one chosen token, which is appended to the text before the next step.

Greedy: always take the top token
The simplest rule picks the highest-probability token every time. It is deterministic and is what "temperature 0" usually means. It works well for extraction, classification and short factual answers. For longer free text it tends to get stuck in loops and bland phrasing, because the single most likely next token is often a safe, generic one.
Sampling with temperature
Instead, draw a token at random according to the probabilities. Temperature T reshapes them first: divide every logit by T, then apply softmax.

- T < 1: more predictable, closer to greedy.
- T = 1: the distribution as the model learned it.
- T > 1: more varied, and more likely to pick something wrong.
Top-k and top-p: cut the tail
Even at a sensible temperature, the thousands of very unlikely tokens together carry real probability, and one bad pick can derail a sentence. Two common filters remove them before sampling:
- Top-k keeps only the k most likely tokens.
- Top-p (nucleus sampling) keeps the smallest set of top tokens whose probabilities add up to at least p.

After filtering, the kept probabilities are renormalised to sum to 1, and one token is sampled. Top-p is usually preferred because a fixed k is too many when the model is sure and too few when many continuations are reasonable. A minimal version in Python:
import math, random
def sample_next(logits, temperature=1.0, top_k=None, top_p=None, rng=random):
if temperature == 0: # greedy
return max(range(len(logits)), key=lambda i: logits[i])
scaled = [x / temperature for x in logits]
order = sorted(range(len(scaled)), key=lambda i: scaled[i], reverse=True)
if top_k is not None:
order = order[:top_k]
m = max(scaled[i] for i in order) # subtract max for stability
weights = {i: math.exp(scaled[i] - m) for i in order}
total = sum(weights.values())
probs = [(i, weights[i] / total) for i in order]
if top_p is not None:
kept, cum = [], 0.0
for i, p in probs:
kept.append((i, p)); cum += p
if cum >= top_p:
break
total = sum(p for _, p in kept)
probs = [(i, p / total) for i, p in kept]
r, cum = rng.random(), 0.0
for i, p in probs:
cum += p
if r < cum:
return i
return probs[-1][0]
With the example probabilities, 10,000 draws with top_k=3 give "hit", "missed" and "cold" about 52%, 34% and 14% of the time, exactly the renormalised values.
Other controls you will see
- Repetition and frequency penalties lower the scores of tokens that already appeared, to fight loops.
- Stop sequences end generation when a given string appears.
- Max tokens caps the length of the answer, and with it cost and latency.
- Seed: some APIs accept one to make sampling repeatable, but results can still vary across hardware and batching.
- Constrained decoding masks out tokens that would break a grammar, for example invalid JSON (see Structured Output).
- Beam search keeps several candidate sequences and picks the best overall; common in translation, rare for chat (see Implement Beam Search).
Trade-offs and costs
| Setting | Gains | Loses |
|---|---|---|
| Greedy / T = 0 | Repeatable, best for exact tasks | Loops, dull long text |
| Low T (0.2–0.5) | Mostly stable, a little variety | Still some randomness |
| T ≈ 0.7–1 with top-p ≈ 0.9–0.95 | Natural, varied writing | Occasional odd choices |
| High T (> 1) | Brainstorming, diversity | Errors and nonsense rise fast |
Sampling settings cost almost nothing to compute; the price is in quality and repeatability. The ranges above are common starting points, not rules; tune them with evals.
In production
- Different tasks, different settings. Use T = 0 (or close) for extraction, classification, tool-call arguments and code edits; use moderate T with top-p for writing and chat.
- Non-determinism is normal. Even at T = 0, batching and floating-point order on GPUs can change outputs slightly. Never rely on byte-identical responses; compare meaning in tests.
- Self-consistency. Sampling several answers and taking the majority can raise accuracy on reasoning tasks, at several times the cost.
- Failure modes. Repetition loops at low temperature; invented facts and broken formats at high temperature; truncated answers when max tokens is too low.
When to reach for it
Tune decoding when outputs are too repetitive, too random, in the wrong format, or when you need repeatable results for tests and audits. Pair it with evals so each change is measured.
Example
The invoice extractor above was running at the API default temperature of 1.0. Setting T = 0 and adding constrained JSON output removed the invented fields. The slogan generator was at T = 0.2; moving it to T = 0.9 with top-p = 0.95 and asking for five options per call fixed the repetition. Same model, two settings.
Common mistakes
- Leaving the default temperature for extraction or tool calls.
- Applying top-k after softmax without renormalising, so probabilities no longer sum to 1.
- Changing temperature and top-p at the same time and not knowing which one helped.
- Expecting T = 0 to be perfectly deterministic across runs and hardware.
- Treating high temperature as "more creative" without limits; past about 1.2 most models degrade quickly.
Say this in the interview
"The model gives a probability for every token; decoding chooses one. Greedy is deterministic but loops on long text. Sampling with temperature divides the logits by T: lower is safer, higher is more varied. Top-k or top-p trims the unlikely tail before sampling; top-p adapts to how confident the model is. I'd use T near 0 for extraction and tool calls, and moderate T with top-p around 0.9 for open writing, and tune both with evals."
Interview questions on this
- What does temperature 0 mean mathematically? Dividing by T → 0 makes the top logit dominate, so softmax becomes one-hot: greedy decoding.
- Top-k or top-p, and why? Top-p, usually: it adapts the number of candidates to the model's confidence.
- Why might T = 0 still give different outputs? Batching and floating-point reduction order on GPUs can change logits slightly, flipping near ties.
- How would you stop a model from repeating itself? Raise temperature a little, add a repetition or frequency penalty, and check the prompt isn't inviting repetition.
- Where should each setting be applied, before or after softmax? Temperature scales logits before softmax; top-k and top-p filter, then renormalise.
Used in
- AI Engineering: Implement Top-k and Top-p Sampling (ML coding); Implement Beam Search; Structured Output; Speculative Decoding; Why LLMs Hallucinate.
- AI System Design: Design ChatGPT; Design a Prompt Playground (exposes these settings).