Two-Tower Models and Vector Search
On this page
The retrieval stage has to find about 500 good candidates out of 10 million items in roughly 10 milliseconds, personalised for each user. A model that takes a user and an item together and outputs a score can't do it: it would need 10 million evaluations per request. What you need is a model whose item side can be computed ahead of time, so that at request time the search becomes "find the items closest to this one vector".
That is what a two-tower model gives you, and approximate nearest-neighbour search is how you query it fast.
How it works
Two towers

- User tower: a neural network that takes user features (ID embedding, recent watches or searches, country, device, time) and outputs a vector
u, say 64 numbers. - Item tower: a separate network that takes item features (ID embedding, text and image embeddings, creator, category, age) and outputs a vector
vof the same size. - Score: the dot product
u · v(or cosine similarity). High means "this user is likely to engage with this item".
The key constraint: the towers never see each other's inputs. That loses some accuracy (no user × item cross features), and it is exactly what allows every item vector to be computed once, offline, and indexed.
Training
Training examples are positive pairs from logs: (user, item they watched, clicked or bought). For each positive, the model needs negatives: items the user didn't choose.
- In-batch negatives: in a batch of 1,024 positive pairs, each user's item is the positive and the other 1,023 items in the batch serve as negatives. Cheap and effective.
- Popularity correction: popular items appear as in-batch negatives far more often, which unfairly pushes them down. A standard fix subtracts the log of each item's sampling probability from its score during training.
- Random and hard negatives: adding random items from the whole catalogue, and some near-misses, sharpens the model.
The loss is usually a softmax over the positive and its negatives: make u · v_positive large relative to u · v_negative. This is the same idea as contrastive learning.
Serving with nearest-neighbour search

- Offline: run the item tower over all 10 million items and build an approximate nearest-neighbour (ANN) index. Rebuild daily, and add new items within minutes so fresh content is retrievable.
- Online: compute the user vector from the latest activity (a few milliseconds), query the index for the top 500 by dot product, and pass them to ranking.
Why approximate? Exact search compares the user vector with every item vector. ANN indexes trade a little recall for a huge speed-up:
- Graph-based indexes (such as HNSW) link each vector to its near neighbours and walk the graph toward the query.
- Clustering-based indexes (inverted file indexes) group vectors into clusters and only search the few clusters closest to the query.
- Quantisation (such as product quantisation) compresses vectors so more fit in memory and comparisons are faster.
Typical settings return most of the true top results (often 90–99% recall) in a few milliseconds. Tune the recall-versus-speed setting with the same recall@k metric you use for the retrieval stage.
Memory: 10 million items × 64 dimensions × 4 bytes ≈ 2.6 GB of raw vectors, which fits on one machine; replicate the index for throughput, and shard it only when the catalogue grows to hundreds of millions.
Trade-offs and costs
| Choice | Gains | Costs |
|---|---|---|
| Two-tower vs cross model | precomputable items, millisecond retrieval | no user × item interaction, less accurate (fine for retrieval) |
| Larger embeddings | more expressive | more memory, slower search |
| ANN vs exact search | orders of magnitude faster | misses a few of the true top items |
| Frequent index rebuilds | fresh items retrievable | compute and operational cost |
| ID embeddings | strong for popular items | useless for new items; add content features |
When to reach for it
The retrieval stage of any large recommendation or search problem: videos, products, people, ads, posts. Also for similarity problems where both sides are the same kind of thing: similar items, visual search, duplicate detection. Then a single tower (one encoder for both sides) is enough.
Example
Prompt: "Design People You May Know."
- User tower (for the person viewing) and candidate tower (for the person suggested) share the same architecture: inputs are the person's ID embedding, school and workplace embeddings, location, and an embedding of their connections.
- Positives: connection requests that were accepted. Negatives: in-batch, plus suggestions shown and ignored as hard negatives.
- Serving: embeddings for all members are indexed daily; new members are added within minutes using only profile features.
- Combined with graph retrieval: friends-of-friends finds people with mutual connections; embedding retrieval finds people from the same school or company with no mutual connections yet. Both feed the ranker.
Common mistakes
- Putting cross features in a tower, which breaks precomputation.
- Using the two-tower score as the final ranking. It is a retrieval model; a ranker with cross features should follow.
- Only in-batch negatives without popularity correction, which over-penalises popular items.
- Stale index: new items invisible until tomorrow's rebuild.
- No recall measurement of the ANN step itself.
Say this in the interview
"For retrieval I'd train a two-tower model on watched pairs with in-batch negatives and a popularity correction. Item vectors are computed offline and put in an approximate nearest-neighbour index, rebuilt daily with new items added within minutes. At request time I compute one user vector from recent activity and fetch the top 500 in a few milliseconds. The towers can't use user-item cross features, which is fine here, because the ranker after it will."