Feature Stores and Training-Serving Skew

Updated Nov 16, 2026 · 4 min read
On this page

    A team trains a ranking model that beats the old one by 3% offline. They launch it and the A/B test shows nothing, then slightly negative. After two weeks of digging they find it: the feature "clicks in the last 7 days" was computed by a daily SQL job for training, counting by calendar day, and by a Java service at serving time, counting a rolling 168 hours and dropping bot traffic. Same name, same idea, different numbers. The model learned on one distribution and was served another.

    That gap has a name: training-serving skew. Feature stores are the most common way to prevent it.

    How it works

    What skew looks like

    Two code paths compute "the same" feature differently. The model trains on one set of values and is served another, and nothing crashes.
    Two code paths compute "the same" feature differently. The model trains on one set of values and is served another, and nothing crashes.

    Skew is any difference between the inputs a model saw in training and the inputs it gets in production. The common causes:

    1. Two implementations. A batch job for training and a service for serving, written in different languages by different people.
    2. Different time. Training reads today's tables for last month's rows (leakage), while serving only knows the past.
    3. Different freshness. Training uses fully computed daily aggregates; serving reads a value that is up to 24 hours stale.
    4. Different defaults. A missing value becomes 0 in one path and the average in the other.
    5. Different filtering. Bots, test accounts or deleted items are removed in one path and not the other.

    Skew is dangerous because nothing breaks. There's no error, just a model that is quietly worse than it should be.

    What a feature store does

    Each feature is defined once. Batch and stream jobs fill an offline store with full history and an online store with the latest values, so training and serving read the same numbers.
    Each feature is defined once. Batch and stream jobs fill an offline store with full history and an online store with the latest values, so training and serving read the same numbers.

    A feature store gives every feature one definition and serves it through two stores:

    • Feature definitions: code, reviewed and versioned like any other code. "clicks_7d for an item: count of click events for this item in the last 168 hours, bots excluded."
    • Batch and stream jobs: run the definitions. Daily batch jobs compute slow-changing features; streaming jobs update fast ones (counts in the last 5 minutes) from an event stream.
    • Offline store: every value with its timestamp, in a data warehouse or data lake. Training jobs do a point-in-time join: for each training row at time t, take each feature's value as it was at t, never later.
    • Online store: only the latest value per entity, in a low-latency key-value store. At request time the serving path reads all features for a user and its candidate items in a few milliseconds.

    Plus the practical parts: a registry so teams can find and reuse features, monitoring of freshness and value distributions, and backfills when a new feature needs history.

    The simpler alternative: log at serving time

    Many teams avoid point-in-time joins altogether by logging features when they are served. When the model scores a candidate, the exact feature vector it used is written to a log along with a request ID. Labels arrive later and are joined by that ID. Training then uses exactly what serving saw: no skew by construction.

    The cost: a new feature has no history until it has been logged for a while, so you either wait a few weeks or backfill it once through the offline store.

    Trade-offs and costs

    ApproachGainsCosts
    Two separate code pathsquick to startskew is almost guaranteed over time
    Feature store with point-in-time joinsone definition, reuse across teams, history for new featuresinfrastructure to run, joins are expensive
    Log features at serving timezero skew by construction, simple trainingnew features need time or a backfill; logs are large
    • Freshness costs money. Streaming features need an event pipeline and a stream processor. Use them only where minutes matter: fraud velocity counters, trending items, courier supply.
    • Online store size. Features for 500 million users × 200 features × 8 bytes is about 800 GB: fine for a sharded key-value store, but worth stating.
    • Latency. One batched lookup per entity type (user, items) keeps reads within a few milliseconds; per-feature lookups do not.

    When to reach for it

    • When the interviewer asks "how do you make sure training and serving features match?"
    • When a design has real-time features (fraud, ETA, ads) next to daily ones.
    • When several models share features (feed, notifications and ads all use user interest embeddings).
    • When asked to debug "great offline, flat online": skew is one of the first three things to check, together with leakage and a label that doesn't match the product metric.

    Example

    Prompt: "Design payment fraud detection." The design needs features like "number of payments from this card in the last hour."

    • Definition: card_txn_count_1h, events from the payment stream, keyed by card, 1-hour sliding window.
    • Streaming job: consumes the payment stream and updates the counter in the online store within seconds.
    • Offline store: the same job also writes each update with its timestamp to the offline store.
    • Training: for each historical payment, a point-in-time join takes card_txn_count_1h as it was just before that payment, so the payment never counts itself, and nothing after it leaks in.
    • Serving: the fraud service reads all card, device and merchant features in one batched lookup, under 10 ms.
    • Monitoring: alert if the online value distribution drifts away from the offline distribution for the same day.

    Common mistakes

    1. "We'll compute features in the service" with no word on how training gets the same values.
    2. Joining labels to current feature tables, which leaks the future into training.
    3. Streaming everything, adding cost and complexity for features that change daily.
    4. No default policy for missing features, so serving and training fill gaps differently.
    5. No freshness monitoring, so a stuck job silently serves week-old values.

    Say this in the interview

    "To avoid training-serving skew, each feature is defined once in a feature store. Batch and streaming jobs write to an offline store with full history and an online store with the latest values. Training uses point-in-time joins so no future data leaks in, and serving reads the online store in one batched call. I'd also log the served feature vectors, which lets me check for skew and train directly on what the model saw."

    Used in