Imbalanced Data and Sampling

Updated Nov 16, 2026 · 4 min read
On this page

    A candidate proposes a fraud model and says it will reach 99.9% accuracy. The interviewer nods and asks: "What does a model that approves every payment score?" Also 99.9%. When positives are rare, accuracy says nothing, the model can learn to ignore the rare class entirely, and the training set is dominated by examples that teach very little.

    Imbalanced data is the normal case in ML system design: fraud, spam, harmful content, ad clicks, purchases. This page covers how to measure, sample and train when the class you care about is 1 in 100 or 1 in 10,000.

    How it works

    Why imbalance hurts

    With one fraud in a thousand payments, a model that never flags anything is 99.9% accurate and catches nothing.
    With one fraud in a thousand payments, a model that never flags anything is 99.9% accurate and catches nothing.

    Three separate problems hide under "imbalance":

    1. The metric lies. Accuracy rewards predicting the majority class.
    2. The loss is dominated by easy negatives. Millions of obviously good payments contribute almost all of the training loss, and the few fraud examples barely move the model.
    3. Training is expensive. Billions of rows, most of which add little, cost compute and slow iteration.

    Fix the metric first

    Use metrics that focus on the rare class:

    • Precision: of the items we flagged, how many were really positive?
    • Recall: of all real positives, how many did we flag?
    • Precision–recall curve and the area under it (PR-AUC), which is far more sensitive than ROC-AUC when positives are rare.
    • Recall at a fixed precision or a fixed false-positive rate: "catch as much fraud as possible while declining no more than 0.1% of good payments." This is usually the number the business cares about.

    Then fix the data

    • Downsample negatives. Keep every positive and a random fraction w of negatives (say 1%). Training is 100 times cheaper and the classes are closer to balanced.
    • Weight the classes. Give each positive a larger weight in the loss instead of (or as well as) dropping negatives.
    • Hard negative mining. Instead of random negatives, keep more of the negatives the model gets wrong or nearly wrong: good payments that look suspicious, posts close to the policy line.
    • Oversample or synthesise positives. Duplicating positives or generating synthetic ones (for example by interpolating between neighbours) is common in textbooks, but in production systems it is used with care: synthetic fraud rarely looks like real fraud.

    Correct the probabilities

    After keeping only 1% of negatives, the model's scores are too high. A simple formula brings them back to real-world probabilities.
    After keeping only 1% of negatives, the model's scores are too high. A simple formula brings them back to real-world probabilities.

    A model trained on downsampled data predicts probabilities for the sample, not for the real world. If you keep a fraction w of negatives, the corrected probability is:

    p = p' / (p' + (1 − p') / w)

    where p' is the model's output. With w = 0.01, a score of 0.50 is really about 0.0099. The order of scores doesn't change, so ranking still works, but anything that uses the probability as a number does need it: thresholds, expected-value calculations, ad auctions, and comparing models trained with different sampling rates.

    Choose a threshold from costs

    A classifier outputs a score; the business needs a decision. Pick the threshold from the costs of each error. If a missed fraud costs $200 on average and a wrongly declined good payment costs $10 in lost margin and goodwill, flag a payment when p × 200 > (1 − p) × 10, which means p > 0.048. Many systems use two thresholds: block above the high one, send to human review between the two, approve below.

    Trade-offs and costs

    TechniqueGainsCosts
    Downsample negativesmuch cheaper training, faster iterationthrows away information; needs calibration correction
    Class weightskeeps all datasame compute cost; large weights can make training unstable
    Hard negative miningteaches the boundary that mattersbiased sample; needs a model to find hard examples
    Oversampling or synthetic positivesmore positive examplesrisk of overfitting or unrealistic examples

    The usual production recipe: downsample easy negatives, keep all positives, add some hard negatives, train, then calibrate on a held-out set that has the true class ratio.

    When to reach for it

    • Fraud, spam, abuse, harmful content, fake accounts: positives well under 1%.
    • Click and conversion prediction: clicks are a few percent of impressions, conversions far fewer.
    • Any interviewer question like "your model has 99% accuracy, is it good?" or "how do you choose the threshold?"

    Example

    Prompt: "Design a spam classifier for email." About 1 in 20 emails is spam on arrival, but in the inbox the class you care about (spam that slipped through) is much rarer.

    • Metric: recall at a false-positive rate of 0.1%, because a legitimate email lost in the spam folder is far worse than one spam in the inbox.
    • Data: keep all user-reported and confirmed spam; downsample normal emails to 10%; add hard negatives: legitimate newsletters and receipts that users rescued from the spam folder.
    • Calibration: correct with w = 0.1, then check calibration on a recent unsampled day of mail.
    • Decision: spam folder above a high threshold, a warning banner between two thresholds, inbox below.

    Common mistakes

    1. Reporting accuracy for a rare-event problem.
    2. ROC-AUC only. It can look excellent while precision at useful thresholds is poor; show the PR curve.
    3. Forgetting calibration after downsampling, then setting a threshold of 0.5 on a model whose scores are inflated 100 times.
    4. Evaluating on the downsampled set. The test set must keep the real class ratio.
    5. Synthetic positives for adversarial problems, which teach the model what fake fraud looks like.
    6. One threshold for everything instead of thresholds chosen from error costs, per segment if needed.

    Say this in the interview

    "Fraud is about 0.1% of payments, so I won't use accuracy. I'll report recall at a false-positive rate the business accepts, and the precision–recall curve. For training I'll keep all fraud and 1% of good payments plus some hard negatives, then correct the probabilities for the sampling rate and calibrate on a recent day with the real ratio. The threshold comes from the cost of a missed fraud versus a declined good customer."

    Used in