Training Pipelines and Retraining
On this page
A recommendation model is trained once, launched, and celebrated. Six months later its click-through rate has fallen 15%. Nobody changed the model; the world changed around it. New videos it has never seen, new users, a holiday season, a competitor's app. A model is a snapshot of the data it was trained on, and that snapshot starts going stale on day one.
In an interview, "I'd train a gradient-boosted model" is half an answer. The other half is how it gets retrained, checked and shipped again, safely and without a human in the loop.
How it works
The pipeline as a series of gates

A production training pipeline runs on a schedule and has the same stages every time:
- Snapshot data. Freeze the training window (for example the last 8 weeks of logged impressions with labels) so the run is reproducible.
- Validate data. Check the schema, the share of missing values, row counts against yesterday, and the label rate. A broken upstream job that silently halves the data should stop the run here, not ship a broken model.
- Train. Usually warm-started from the previous model for daily runs; from scratch for periodic full retrains or when features change.
- Evaluate against the current model. Same test set, same metric, broken down by important segments (new users, each country, each platform). The new model must win overall and must not lose badly on any big segment.
- Register. Store the model with its training data range, code version, feature list and metrics, so you can always answer "what exactly is running?"
- Shadow and canary. Shadow mode: the new model scores live traffic but its outputs are only logged, to check latency and score distributions. Canary: it serves 1–5% of real traffic while online metrics and errors are watched.
- Full rollout, with the previous model kept ready for instant rollback.
Split by time

Production models predict the future from the past, so the offline test must do the same. Train on weeks 1–8, tune on week 9, test on week 10. A random split mixes future rows into training: features like "item popularity" leak, and the offline metric is better than anything the model will achieve in production.
How often to retrain
| Cadence | Typical problems | How |
|---|---|---|
| Minutes to hours | ads click-through, trending content, fraud velocity | online or incremental updates on a stream of new examples |
| Daily | feed and video ranking, recommendations | warm start from yesterday's model on the newest data |
| Weekly to monthly | ETA, search relevance, pricing | full retrain from scratch |
Choose by measuring: train a model, then evaluate it on each following day's data. If quality drops noticeably after two days, retrain daily. Also retrain on events: a new feature, a policy change, or a drift alert.
Scale
When data no longer fits on one machine, the standard tools are data parallelism (each worker trains on a slice of each batch and gradients are averaged) and, for recommendation models whose ID embedding tables reach hundreds of gigabytes, sharding the embedding tables across machines (model parallelism for the embeddings, data parallelism for the rest). Say this in one or two sentences; interviewers rarely want more unless the role is ML infrastructure.
Trade-offs and costs
- Fresher models versus cost and risk. Every retrain costs compute and is a chance to ship a bad model. Frequent retraining only pays if the data changes fast.
- Warm start versus from scratch. Warm starts are cheap and keep learned embeddings stable; from-scratch retrains avoid accumulating old quirks and are required when the feature set changes.
- Automatic versus human-approved deployment. Full automation needs strong gates. Many teams automate daily retrains of an existing model and require a human for any change in features or architecture.
- Long versus short training windows. More history gives more data; recent data reflects today's behaviour. Weighting recent examples more is a common middle ground.
When to reach for it
Step 4 of every ML system design answer, after the model ladder. It is also the answer to "how do you keep the model up to date?", "how do you deploy a new model safely?" and "the model got worse after the last retrain, what happened?"
Example
Prompt: "Design the video recommendation system." Training plan:
- Data: the last 28 days of logged impressions with features and labels (watch at least half). Labels are complete after the session ends, so yesterday's data is usable this morning.
- Validation: fail the run if yesterday's impression count is more than 20% off the 7-day average, or the label rate moves by more than 30%.
- Training: daily warm start of the ranking model; full retrain weekly; two-tower retrieval model retrained weekly, with the item index rebuilt daily for new videos.
- Gate: the new ranker must match or beat the current one on NDCG@10 for yesterday, overall and for new users.
- Rollout: 1 hour of shadow traffic, then 5% canary for 6 hours, then 100%. Automatic rollback if watch time per session drops more than 1% in the canary.
Common mistakes
- Train once, ship forever. No retraining plan at all.
- Random splits on time-ordered data.
- No data validation, so a broken upstream job produces a broken model that passes evaluation on broken test data.
- Comparing against nothing: evaluating the new model alone instead of against the model in production on the same data.
- Ignoring segments: an overall win that hides a large loss for new users or a small country.
- No rollback path.
Say this in the interview
"I'd retrain the ranker daily, warm-started from yesterday, on a 28-day window split by time. The pipeline validates the data first, then trains, then compares against the production model on the same test set and key segments. If it wins, it goes to the registry, runs in shadow, then a 5% canary watching watch time and latency, then full traffic, with the old model ready for rollback. All automatic; a human only approves feature or architecture changes."