Framing a Product Problem as an ML Problem
On this page
A team ships a model that predicts clicks on the home screen. Offline accuracy is excellent. Two months later, clicks are up 8%, but time spent is down and the app store reviews say "full of clickbait". Nothing in the model was wrong. The model learned exactly what it was asked to learn. The question was wrong.
Framing is the step where you decide what the model is asked. In an interview it takes five minutes, and it decides whether everything after it makes sense.
How it works

Step 1: state the business goal and a guardrail
One sentence on what should go up, and one on what must not go down. "More people find videos they enjoy, without more of them reporting content or leaving." The guardrail matters as much as the goal: it is how you catch a model that games the goal.
Step 2: pick the product metric
The goal is too vague to measure, so name the number the company already tracks: daily active users, watch time per day, purchases per session, fraud losses in dollars. This becomes the primary metric of your A/B test later.
Step 3: choose the ML task
Most interview problems map to one of four task types:
| Task | Output | Examples |
|---|---|---|
| Binary classification | a probability for yes or no | spam, fraud, harmful content, will this user click |
| Multi-class or multi-label | a probability per class | which policy a post violates, which topics a video covers |
| Regression | a number | trip duration, price, watch time |
| Ranking | an ordered list | feed, search results, recommendations |
Ranking problems are almost always solved by pointwise scoring in production: predict a probability or value for each candidate, then sort. That turns ranking into classification or regression, which is easier to train, calibrate and debug. Pairwise and listwise losses exist and can help, but start pointwise and say why.
Step 4: define the prediction precisely
Write it as a conditional probability or expectation, with its inputs: P(watch ≥ 50% | user, video, context). This forces three decisions: the event, the unit (per impression, per user–item pair, per transaction) and the time window.
Step 5: define the label
The label is how you will observe that event in your logs. "1 if the user watched at least half of the video in the same session, else 0. Impressions the user scrolled past in under one second are dropped, because they were never really seen." Say how long you wait for the label: immediately for clicks, days for purchases, up to 60 days for chargebacks.
Trade-offs and costs

- Fast labels versus true labels. Clicks arrive in seconds and there are billions of them; satisfaction surveys arrive rarely and are noisy. Fast labels let you retrain often but drift from what you want. Most real systems mix several.
- One objective versus several. A single label is simple. Several labels (click, watch, like, share, report) combined into one score follow value better, but the weights are a product decision you must tune with A/B tests.
- Direct versus proxy for long-term goals. "Is this user still active in 30 days" is the real goal but a slow, noisy label. Short-term proxies that correlate with it are the usual compromise; check the correlation with long-term holdout experiments.
- Model versus rules. If a rule gets 90% of the value (block payments over a limit from new devices), say so. Framing includes deciding whether ML is needed at all.
When to reach for it
Always, at the start of every ML system design answer. It is also the right first move when an interviewer asks "our model's offline metrics are great but the product metric didn't move, why?" The usual answer is that the label or objective does not match the product metric.
Example
Prompt: "Design notifications for a social app."
- Business goal: people return to the app because of notifications they value. Guardrail: notification opt-out rate must not rise.
- Product metric: daily active users; secondary, sessions started from a notification.
- ML task: binary classification per candidate notification, used to decide send or not send.
- Prediction:
P(open within 1 hour | user, notification, time)andP(disable notifications within 1 day | ...). - Decision: send when
P(open) × value − λ × P(disable) > threshold, with a daily cap per user. - Label: opens are logged by the app; opt-outs are logged by settings changes. Both are available within a day, so daily retraining works.
Five lines, and the rest of the design follows from them.
Common mistakes
- Jumping to the model. "I'll use a transformer" before the label exists.
- A label that rewards the wrong thing. Clicks for a feed, or "time in app" for a mental-health feature.
- No guardrail. Every objective can be gamed; name what would show it.
- Mixing units. Training on user-level labels and serving per-impression scores, or the reverse.
- Forgetting the decision. A probability is not an action. Say what threshold or ranking turns it into one.
- Ignoring label delay. A 60-day label cannot drive daily retraining without a faster proxy.
Say this in the interview
"The business wants more people to enjoy their videos without more reports. I'll measure that with daily watch time and use report rate as a guardrail. I'll frame it as ranking, solved pointwise: predict the probability that the user watches at least half of each candidate video, plus smaller heads for likes and reports, and combine them into one score. Labels come from playback logs within the session, so I can retrain daily."
Used in
- Next: Labels and Training Data
- Related: Online Metrics and A/B Testing, The ML System Design Answer Framework
- Problems where framing is the hard part: Design a News Feed Ranking System, Design Notification Ranking, Design Dynamic Pricing