Labels and Training Data

Updated Nov 16, 2026 · 4 min read
On this page

    A fraud model looks brilliant offline: 0.98 AUC. In production it catches almost nothing. The cause takes a week to find: one feature was "number of disputes on this card", which the training job read from today's table. In training, every fraudulent payment already had its dispute recorded. At checkout, none of them do. The model had learned to read the answer key.

    Most ML failures in production are data failures, and most of those are label failures: the wrong label, a label that leaks, or a label that arrives too late. This page covers where labels come from, how to build a training row correctly, and the biases you should name in an interview.

    How it works

    Where labels come from

    Implicit feedback gives volume, explicit feedback gives signal from some users, human review gives truth. Each has its own delay and bias.
    Implicit feedback gives volume, explicit feedback gives signal from some users, human review gives truth. Each has its own delay and bias.
    1. Implicit feedback: what users do. Clicks, watch time, purchases, add-to-cart, dwell time on a result. It is free and enormous, but it only exists for items you chose to show, and it measures attention, not satisfaction.
    2. Explicit feedback: what users say. Likes, dislikes, star ratings, "not interested", reports. Much sparser and skewed toward users who bother to give it, but closer to what people actually value.
    3. Human review: trained raters or domain experts label samples by a written guideline. Expensive and slow, but the closest thing to truth. Essential for trust and safety, search relevance and anything where users can't label for you.

    Other sources come up in specific problems: outcomes from the business (chargebacks for fraud, actual arrival times for ETA), weak labels from rules or older models, and model-generated labels (a large model labels data to train a small, fast one, checked against a human-labelled sample).

    Building one training row

    Features must only use data from before the prediction time; the label comes from a window after it. Logging features at serving time removes the most common leak.
    Features must only use data from before the prediction time; the label comes from a window after it. Logging features at serving time removes the most common leak.

    Every training row answers: at the moment we had to decide, what did we know, and what happened next?

    • The unit: one impression, one transaction, one user–item pair. It must match what the model scores in production.
    • The prediction time t: when the decision was made.
    • Features: computed only from data available before t.
    • Label window: how long after t you watch for the outcome. Ten minutes for a click, a session for a watch, 7 days for a purchase, 60 days for a chargeback.

    The safest way to get features "as of t" is to log them at serving time: when the model scores a candidate, write the exact feature values it used to a log, and later join the label to that log. The alternative is a point-in-time join against timestamped feature snapshots. Reading today's tables when building yesterday's rows is how leaks happen.

    Biases to name

    • Selection bias: you only see outcomes for what you showed or approved. A fraud model never learns whether blocked payments were really fraud.
    • Position bias: items at the top get clicked more because they are at the top, not because they are better.
    • Delayed labels: recent examples look negative because their outcome hasn't happened yet. A purchase made on day 6 turns a "negative" into a positive.
    • Label noise: accidental clicks, raters who disagree, users who report things they merely dislike.
    • Survivorship: users who left stop generating data, so the data overrepresents happy users.

    Trade-offs and costs

    ChoiceGainsCosts
    Short label windowfresh data, frequent retrainingmisses slow outcomes, more false negatives
    Long label windowmore complete labelsstale training data, slower to react to change
    Implicit labelshuge volume, no costbiased toward what was shown, measures attention
    Human labelsaccurate, follow policycost per label, slow, rater disagreement
    Negative samplingsmaller, faster trainingprobabilities must be recalibrated

    A common compromise for slow labels: train on a fast proxy (add-to-cart within 1 day), and periodically check that it still predicts the slow outcome (purchase within 30 days). For fraud, train on data old enough for chargebacks to arrive, and use a separate fast model or rules for brand-new attack patterns.

    When to reach for it

    Step 2 of every ML system design answer. Spend more time here when:

    • the label is rare (fraud, harmful content), so every positive matters,
    • the label is slow (purchases, chargebacks, churn),
    • the label is subjective (relevance, "harmful"), so you need a guideline and raters,
    • the model controls what you observe (blocking, ranking), so selection bias is built in.

    Example

    Prompt: "Design harmful content detection." Labels are the hard part.

    • Sources: user reports (fast, noisy: many reports are disagreement, not violations), reviewer decisions on reported posts (accurate, but only for reported posts), and a random sample of all posts sent to reviewers each day (small, but unbiased: it shows what users never report).
    • Label definition: a post is positive if a trained reviewer, following the written policy, marks it as violating. Reports alone are a feature, not a label.
    • Window: most reports arrive in the first 24 hours; review takes up to 48 more. Rows enter training after 3 days.
    • Biases: reported posts overrepresent popular posts; the random sample corrects for that. Reviewer agreement is measured weekly; categories where agreement is low get a clearer guideline before more labels are bought.

    Common mistakes

    1. Leakage through features that are only known after the outcome.
    2. Random train/test splits on time-ordered data; always split by time.
    3. Treating unlabelled as negative too early, before the label window closes.
    4. Training only on what the current model chose, which teaches the next model to repeat the current one's mistakes.
    5. No guideline for human labels, so raters disagree and the model learns noise.
    6. Different units in training and serving (user-level labels, impression-level scoring).

    Say this in the interview

    "Each training row is one impression with the features we logged at serving time, so there's no leakage. The label is a watch of at least half the video within the session, so the window is short and I can retrain daily. Clicks are biased by position, so I'll log the position and use it in training. For truth I'll also send a small random sample to human raters and track how well our implicit labels agree with them."

    Used in