Model Serving: Batch vs Online
On this page
A candidate designs a recommendation model that scores every one of 10 million videos for each user, with a 400 ms neural network per video. The interviewer asks how long the home page takes to load. The honest answer is "about 46 days". Serving is where the model meets the latency budget, the cost budget and the real world, and where many otherwise good answers fall apart.
How it works
Batch or online

Batch (offline) prediction: a scheduled job scores everything ahead of time and writes results to a key-value store. The app just looks them up.
- Good for: daily "recommended for you" emails, a precomputed list of similar items per item, nightly risk scores for every account.
- Pros: cheap, simple, no latency risk at request time, easy to inspect before anyone sees it.
- Cons: stale (computed hours ago), and only covers users and items that existed when the job ran.
Online (real-time) prediction: the model runs when the request arrives, with fresh context: what the user just did, the time, the query they typed.
- Good for: search ranking, feed ranking, fraud checks at checkout, ETA, anything that depends on the request itself.
- Pros: uses the freshest signals; works for anything, including new users and new items.
- Cons: must fit a latency budget; servers must run all the time for peak load.
Most real systems mix both. Embeddings, candidate lists and slow features are computed in batch; the final ranking runs online. Say this explicitly in the interview.
The latency budget

Start from the product's latency target and divide it between stages. A typical ranked page might allow about 200 ms at p99:
- Gateway and network: about 20 ms
- Retrieval (narrow millions to a few hundred): about 30 ms
- Feature fetch for the user and every candidate: about 30 ms
- Ranking a few hundred candidates with the heavy model: about 80 ms
- Business rules, diversity, re-ranking: about 15 ms
- Spare for tail latency: about 25 ms
The numbers are an example; the habit is what matters. Once each stage has a budget, the design choices follow: the heavy model can only see a few hundred items, so you need a cheap retrieval stage before it.
Making online inference fast
- Cut candidates first: a multi-stage funnel (retrieval → light ranker → heavy ranker), so the expensive model scores hundreds, not millions.
- Batch inside the request: score all candidates in one model call; GPUs and vectorised CPUs are far more efficient on batches.
- Smaller models: distil a large model into a smaller one, quantise weights to 8-bit, prune features that add little.
- Cache: cache user embeddings for a few minutes and item features for longer; cache whole results for logged-out traffic.
- Precompute what doesn't depend on the request: item embeddings and item-side towers run in batch.
- Timeouts and fallbacks: if the ranker is slow or down, serve the retrieval order, or a popular-items list, rather than an error.
The serving stack
A model server (a service that loads the model and answers scoring requests) sits behind the API gateway and load balancer with the rest of the backend. Models are loaded from the model registry; new versions roll out like any deployment (shadow, canary, full) with the previous version kept for rollback. Every request logs the features used and the scores produced, for training and monitoring.
Trade-offs and costs
| Choice | Gains | Costs |
|---|---|---|
| Batch prediction | cheap, simple, no request-time risk | stale; no new users or items until the next run |
| Online prediction | fresh, covers everything | latency budget, always-on compute |
| Bigger ranking model | better quality | fewer candidates per request, or more hardware |
| GPU serving | fast for large neural models | cost; only worth it with batching and steady load |
| CPU serving | cheap, simple | fine for trees and small networks; slow for big ones |
| Aggressive caching | lower latency and cost | staleness; memory |
Cost is an interview-worthy number. Example: 50,000 requests a second × 500 candidates = 25 million model evaluations a second. If one CPU core scores 20,000 candidates a second with a tree model, that's about 1,250 cores. Doubling the candidates doubles the bill.
When to reach for it
Step 5 of the framework. Go deep when the prompt mentions a strict latency ("autocomplete under 50 ms", "fraud check during checkout"), very large catalogues, or when the interviewer asks "how does this run in production?" or "what happens if the model service is down?"
Example
Prompt: "Design ETA prediction for a ride-hailing app."
- Online: the ETA must reflect current traffic and the exact pickup and drop-off, so the model runs per request.
- Budget: the price-and-ETA screen needs both within about 300 ms; the ETA model gets 50 ms.
- Batch parts: road-segment speed profiles by hour of week are precomputed nightly; live segment speeds update every minute from a stream.
- Model: a routing engine computes a route and a physics-based estimate; a gradient-boosted model corrects it using live traffic, time, weather and pickup features. Trees run fast on CPU.
- Fallback: if the model times out, show the routing engine's estimate.
Common mistakes
- Scoring everything with the heavy model, with no retrieval stage.
- No latency budget, so there is no way to judge any choice.
- Batch-only for a problem that depends on the request (search, fraud).
- No fallback when the model service fails.
- Ignoring cost: GPUs for a tree model, or no estimate of the hardware needed.
- Not logging served features, which loses the training data and the skew check.
Say this in the interview
"Ranking runs online because it depends on the user's latest activity, but everything that doesn't, like item embeddings and candidate indexes, is built in batch. With a 200 ms budget, retrieval gets about 30 ms to narrow millions of items to 500, feature fetch about 30, and the ranker about 80 ms to score those 500 in one batched call. If the ranker times out we serve the retrieval order, and every served feature vector is logged."
Used in
- Before this: Online Metrics and A/B Testing
- Next: Monitoring and Data Drift
- Related: Retrieval and Ranking: the Multi-Stage Funnel, Feature Stores and Training-Serving Skew, Two-Tower Models and Vector Search
- Problems: Design ETA Prediction, Design Query Autocomplete, Design a Video Recommendation System