Retrieval and Ranking: the Multi-Stage Funnel
On this page
You have 10 million videos and 200 milliseconds. Your best ranking model takes half a millisecond per video. Scoring everything would take 5,000 seconds per page load. Using a model small enough to score everything in time would give poor recommendations. Neither works.
The answer used by almost every large recommendation, search and ads system is to split the work into stages: cheap models look at many items, expensive models look at few. This is the single most important pattern in ML system design, and interviewers expect it in any ranking problem.
How it works
The funnel

- Retrieval (candidate generation): from millions of items to about 1,000, in a few tens of milliseconds. Methods are cheap per item: nearest-neighbour search on embeddings, lookups in precomputed lists ("items often watched after this one"), the user's followed accounts, trending lists. Retrieval is judged by recall: did the good items make it into the 1,000?
- Light ranking (optional): from about 1,000 to about 200 with a small model and a few features. Used when the heavy ranker can't afford 1,000 items.
- Heavy ranking: scores a few hundred items with the best model you have: hundreds of features, cross features, sometimes several predicted outcomes at once (click, watch time, like, report). Judged by precision at the top, for example NDCG@10.
- Re-ranking and business rules: the final ordering is adjusted for things a pointwise model can't see: diversity (not ten videos from one creator), freshness, already-seen items, policy filters, ad slots. Usually rules or a light optimisation.
Many retrieval sources

No single retrieval method finds everything a user might like, so most systems run several in parallel and merge the results:
- Two-tower embeddings + approximate nearest neighbour (ANN) search: personal taste, learned from behaviour.
- Item-to-item co-occurrence: "people who watched what you just watched also watched these".
- Graph sources: creators you follow, friends' activity.
- Popular and fresh: trending in your country, new uploads, which also helps cold start.
- Query matching for search: an inverted index on text, plus embedding search for meaning.
Each source gets a quota (for example 400 from embeddings, 200 from co-occurrence, 200 from follows, 200 fresh), results are de-duplicated, and the merged set goes to ranking. Quotas stop one source from crowding out the rest and keep the ranker's input varied.
Why stages need different models
- Retrieval must be precomputable. The item side can't depend on the user, so item embeddings are computed in batch and indexed; at request time you only compute one user embedding and search. That's why retrieval models are "two-tower": user and item never interact until a dot product.
- Ranking can use interactions. With only hundreds of candidates, the ranker can use features that combine user and item (has this user watched this creator?), which are the most predictive and impossible to precompute for all pairs.
- Each stage is trained for its job. Retrieval is trained to separate plausible items from random ones (many negatives, recall-oriented); ranking is trained on actual impressions to order plausible items precisely.
Trade-offs and costs
| Decision | More | Less |
|---|---|---|
| Candidates from retrieval | better recall, higher ranking cost | faster, risks missing good items |
| Number of stages | each stage cheaper, more tuning | simpler system, less room for a big ranker |
| Retrieval sources | variety, cold-start coverage | more systems to run and debug |
| Re-ranking rules | product control (diversity, policy) | can undo what the ranker learned |
- Stage mismatch: a ranker can only pick from what retrieval gives it. If retrieval misses an item, nothing downstream can fix it. Monitor retrieval recall separately.
- Training data bias: the ranker is trained only on items that were shown, which were chosen by the previous system. Exploration and logging position help (see Feedback Loops).
When to reach for it
Any problem where the number of items is much larger than what one model can score within the latency budget: video, feed, products, people you may know, search, ads, notifications, similar items. Draw it in the high-level design, give each stage a size and a latency, then go deep on whichever stage the interviewer picks.
Example
Prompt: "Design product recommendations for an online store's home page," 50 million products, 150 ms budget.
- Retrieval (about 25 ms): 300 from a two-tower model on the user's recent views and purchases; 200 "bought together" with recent purchases; 100 from categories the user browsed today; 100 trending in their region. Merge to about 650 after de-duplication.
- Light ranker (about 15 ms): a small gradient-boosted model with 30 features keeps the top 150.
- Heavy ranker (about 50 ms): a neural model predicting click and purchase, with user × product cross features; score = P(click) × P(purchase | click) × margin weight.
- Re-rank (about 5 ms): at most 3 products per brand in the top 20, remove out-of-stock items and items bought in the last 30 days, insert 2 sponsored slots.
Common mistakes
- One model for everything, either too slow or too weak.
- No sizes or latencies per stage, so the funnel is a picture with no reasoning.
- Ranking features in retrieval: user × item crosses can't be precomputed for 10 million items.
- A single retrieval source, usually only embeddings, which misses fresh items and social signals.
- Measuring only the final ranking, never retrieval recall.
- Forgetting re-ranking, so the page is ten near-identical items.
Say this in the interview
"With 10 million items and 200 ms, I'll use a funnel. Retrieval pulls about a thousand candidates from several sources: two-tower embeddings with nearest-neighbour search, co-watched items, followed creators and trending. A heavy ranker with cross features scores those and predicts watch, like and report, and a re-ranking step enforces diversity and policy. I'll measure retrieval by recall at 1,000 and the ranker by NDCG at 10."