Fundamentals

  1. 1 Framing a Product Problem as an ML Problem "The business wants more people to enjoy their videos without more reports. I'll measure that with daily watch time and use report rate as a guardrail. I'll frame it as ranking, solved pointwise: predict the probability that the user watches at least half of each candidate video, plus smaller hea... 4 min read
  2. 2 Labels and Training Data "Each training row is one impression with the features we logged at serving time, so there's no leakage. The label is a watch of at least half the video within the session, so the window is short and I can retrain daily. Clicks are biased by position, so I'll log the position and use it in traini... 4 min read
  3. 3 Features and Embeddings "I'll group features as user, item, context and user-item cross. The cross features, like how often this user finished this creator's videos, will carry most of the signal. IDs become learned embeddings, counts are taken over several time windows, and rates are smoothed. Most features update dail... 4 min read
  4. 4 Feature Stores and Training-Serving Skew "To avoid training-serving skew, each feature is defined once in a feature store. Batch and streaming jobs write to an offline store with full history and an online store with the latest values. Training uses point-in-time joins so no future data leaks in, and serving reads the online store in on... 4 min read
  5. 5 Imbalanced Data and Sampling "Fraud is about 0.1% of payments, so I won't use accuracy. I'll report recall at a false-positive rate the business accepts, and the precision–recall curve. For training I'll keep all fraud and 1% of good payments plus some hard negatives, then correct the probabilities for the sampling rate and ... 4 min read
  6. 6 Training Pipelines and Retraining "I'd retrain the ranker daily, warm-started from yesterday, on a 28-day window split by time. The pipeline validates the data first, then trains, then compares against the production model on the same test set and key segments. If it wins, it goes to the registry, runs in shadow, then a 5% canary... 4 min read
  7. 7 Offline Metrics: Precision, Recall, AUC, NDCG "For retrieval I'll measure recall at 500, because that stage only needs to get the right items into the candidate set. For ranking I'll use NDCG at 10 with graded labels from purchases, carts and clicks. Since the score also feeds the ads blend, I'll check calibration too. A new model must beat ... 5 min read
  8. 8 Online Metrics and A/B Testing "Offline wins only earn the model an A/B test. I'd randomise by user, 50/50 after a 1% ramp, with watch time per user as the primary metric and reports, hides and latency as guardrails. To detect a 1% lift we need a few million users per arm, so two weeks covers the sample and lets novelty fade. ... 5 min read
  9. 9 Model Serving: Batch vs Online "Ranking runs online because it depends on the user's latest activity, but everything that doesn't, like item embeddings and candidate indexes, is built in batch. With a 200 ms budget, retrieval gets about 30 ms to narrow millions of items to 500, feature fetch about 30, and the ranker about 80 m... 4 min read
  10. 10 Monitoring and Data Drift "Models fail quietly, so I'd monitor four layers: system health, input data, the prediction distribution, and outcomes once labels arrive, each broken down by key segments. Input and score checks are the early warning because labels lag. Data drift shows up in feature distributions; concept drift... 4 min read