Implement Top-k and Top-p Sampling
On this page
Problem statement
Implement
sample_next_token(logits, temperature, top_k, top_p)in NumPy.logitsis a 1-D array of raw scores, one per vocabulary token. Apply temperature, then keep only thetop_khighest-scoring tokens (0 means no limit), then keep the smallest set of the remaining tokens whose probabilities add up to at leasttop_p(1.0 means no limit), renormalise, and return the token id of one random draw. Temperature 0 means greedy decoding.
This is one of the most frequently reported ML coding tasks for LLM roles. It looks simple; the marks are in the order of operations, the edge cases, and returning the right index.
Example

probabilities: hit 0.45 missed 0.30 cold 0.12 full 0.08 purple 0.05
temperature=0 → always "hit"
top_k=3 → hit 0.52, missed 0.35, cold 0.14 (renormalised)
top_p=0.9 → hit 0.47, missed 0.32, cold 0.13, full 0.08
Constraints (assumed)
- Vocabulary size V up to about 200,000; one call per generated token, so it must be vectorised.
- 0 ≤ temperature; 0 ≤ top_k ≤ V; 0 < top_p ≤ 1.
- A random generator can be passed in for reproducible tests.
What this question tests
- Knowing what each knob does and the order in which to apply them.
- Numerical stability (shift before exp) and renormalising after filtering.
- Correct index bookkeeping between sorted positions and token ids.
- Handling edge cases: temperature 0, top_p very small, ties.
Questions to ask first
- What should temperature 0 do? (Greedy: argmax.)
- If both top-k and top-p are set, which applies first? (Common convention: top-k, then top-p.)
- Does top-p include the token that crosses the threshold? (Yes: keep the smallest set whose sum is ≥ p.)
- Single sequence or a batch? (Single here; batching is a follow-up.)
- Any repetition penalty or banned tokens? (Not here.)
Try it yourself first
You have the full problem. Sketch your answer, then sign up free to read the complete walkthrough.
- The complete design, flow by flow, with pictures
- Deep dives into the hard parts
- What mid, senior and staff candidates are expected to say
- Full company question lists
Free forever. No card needed.