Framing a Product Problem as an ML Problem

Updated Nov 16, 2026 · 4 min read
On this page

    A team ships a model that predicts clicks on the home screen. Offline accuracy is excellent. Two months later, clicks are up 8%, but time spent is down and the app store reviews say "full of clickbait". Nothing in the model was wrong. The model learned exactly what it was asked to learn. The question was wrong.

    Framing is the step where you decide what the model is asked. In an interview it takes five minutes, and it decides whether everything after it makes sense.

    How it works

    Five steps from what the business wants to a label a model can learn. Each step is easier to measure and a little further from the real goal.
    Five steps from what the business wants to a label a model can learn. Each step is easier to measure and a little further from the real goal.

    Step 1: state the business goal and a guardrail

    One sentence on what should go up, and one on what must not go down. "More people find videos they enjoy, without more of them reporting content or leaving." The guardrail matters as much as the goal: it is how you catch a model that games the goal.

    Step 2: pick the product metric

    The goal is too vague to measure, so name the number the company already tracks: daily active users, watch time per day, purchases per session, fraud losses in dollars. This becomes the primary metric of your A/B test later.

    Step 3: choose the ML task

    Most interview problems map to one of four task types:

    TaskOutputExamples
    Binary classificationa probability for yes or nospam, fraud, harmful content, will this user click
    Multi-class or multi-labela probability per classwhich policy a post violates, which topics a video covers
    Regressiona numbertrip duration, price, watch time
    Rankingan ordered listfeed, search results, recommendations

    Ranking problems are almost always solved by pointwise scoring in production: predict a probability or value for each candidate, then sort. That turns ranking into classification or regression, which is easier to train, calibrate and debug. Pairwise and listwise losses exist and can help, but start pointwise and say why.

    Step 4: define the prediction precisely

    Write it as a conditional probability or expectation, with its inputs: P(watch ≥ 50% | user, video, context). This forces three decisions: the event, the unit (per impression, per user–item pair, per transaction) and the time window.

    Step 5: define the label

    The label is how you will observe that event in your logs. "1 if the user watched at least half of the video in the same session, else 0. Impressions the user scrolled past in under one second are dropped, because they were never really seen." Say how long you wait for the label: immediately for clicks, days for purchases, up to 60 days for chargebacks.

    Trade-offs and costs

    A click label rewards clickbait. A label closer to real value, or a weighted mix of several, keeps the model honest.
    A click label rewards clickbait. A label closer to real value, or a weighted mix of several, keeps the model honest.
    • Fast labels versus true labels. Clicks arrive in seconds and there are billions of them; satisfaction surveys arrive rarely and are noisy. Fast labels let you retrain often but drift from what you want. Most real systems mix several.
    • One objective versus several. A single label is simple. Several labels (click, watch, like, share, report) combined into one score follow value better, but the weights are a product decision you must tune with A/B tests.
    • Direct versus proxy for long-term goals. "Is this user still active in 30 days" is the real goal but a slow, noisy label. Short-term proxies that correlate with it are the usual compromise; check the correlation with long-term holdout experiments.
    • Model versus rules. If a rule gets 90% of the value (block payments over a limit from new devices), say so. Framing includes deciding whether ML is needed at all.

    When to reach for it

    Always, at the start of every ML system design answer. It is also the right first move when an interviewer asks "our model's offline metrics are great but the product metric didn't move, why?" The usual answer is that the label or objective does not match the product metric.

    Example

    Prompt: "Design notifications for a social app."

    • Business goal: people return to the app because of notifications they value. Guardrail: notification opt-out rate must not rise.
    • Product metric: daily active users; secondary, sessions started from a notification.
    • ML task: binary classification per candidate notification, used to decide send or not send.
    • Prediction: P(open within 1 hour | user, notification, time) and P(disable notifications within 1 day | ...).
    • Decision: send when P(open) × value − λ × P(disable) > threshold, with a daily cap per user.
    • Label: opens are logged by the app; opt-outs are logged by settings changes. Both are available within a day, so daily retraining works.

    Five lines, and the rest of the design follows from them.

    Common mistakes

    1. Jumping to the model. "I'll use a transformer" before the label exists.
    2. A label that rewards the wrong thing. Clicks for a feed, or "time in app" for a mental-health feature.
    3. No guardrail. Every objective can be gamed; name what would show it.
    4. Mixing units. Training on user-level labels and serving per-impression scores, or the reverse.
    5. Forgetting the decision. A probability is not an action. Say what threshold or ranking turns it into one.
    6. Ignoring label delay. A 60-day label cannot drive daily retraining without a faster proxy.

    Say this in the interview

    "The business wants more people to enjoy their videos without more reports. I'll measure that with daily watch time and use report rate as a guardrail. I'll frame it as ranking, solved pointwise: predict the probability that the user watches at least half of each candidate video, plus smaller heads for likes and reports, and combine them into one score. Labels come from playback logs within the session, so I can retrain daily."

    Used in