Feature Stores Explained: Same Features in Training and Serving

The model quietly makes worse decisions, because the inputs it sees live no longer mean what they meant in training. Nobody changed the model, and usually no error appears.

A feature is one input number or label the model uses to decide, such as how many loan payments a person missed. A feature store exists to keep those inputs identical in both places.

Picture a common setup. A lending app gives instant two-wheeler loans: a buyer in a showroom applies on a phone and gets a yes or no within seconds. The setup is made up, but the problem is real.

A data scientist builds a model that predicts default risk. That is the chance a borrower stops paying their EMIs, the fixed monthly loan instalments. A high score means the app should say no.

She picks about 40 features and builds training data in a notebook, using SQL on the company's data warehouse, a large database built for analysis. Training means showing the model 18 months of past applications and how each loan turned out.

A platform engineer then puts the model behind the loan API. He rewrites each of the 40 features in the API's Python code, reading from the live database. This live side is called serving.

Three months later, defaults on new loans climb from 2.1% to 3.4% (made-up numbers). The two of them take 1,000 recent applications and compute every feature both ways, side by side. Three mismatches jump out.

• Average monthly inflow, the money coming into the applicant's bank account. The notebook averaged 90 days of UPI credits, payments made through India's instant payment system. The API averaged 30 days. Same name, different number.
• Missed EMI count. The notebook counted every missed EMI in today's table, including misses that happened after the application. The model had learned from the future.
• City. The notebook stored "Ahmedabad", but the API sent "AHMEDABAD". To the model, the capital letters looked like a city it had never seen.

None of these is a model bug. Each one is the same feature written twice, by two people, in two places. The rest of this guide fixes that with a feature store, and shows when one is more than you need.

A feature store is a shared system that defines each feature once, computes it, stores its values, and serves those same values to model training and to live predictions. It sits between your raw data and your models.

Picture a kushti tournament, the traditional Indian mud wrestling. Wrestlers compete in weight classes. At the trials, each one steps on the official scale at the akhada, the wrestling ground, early in the morning, before eating. On bout day, the same scale is used under the same rule.

Now imagine the trials used a bathroom scale after lunch, and bout day used the official scale at dawn. A wrestler could train for months in the wrong class. The scale was never the problem. Two ways of weighing were.

The mapping is direct. The official scale is the shared feature definition. The morning rule is the time window. The trials are training, and bout day is serving. Where the picture breaks: a scale weighs once, but a feature store must also remember every past value and serve the latest in milliseconds.

Most feature stores have four parts. Each one fixes a piece of the loan app's problem.

• Feature definitions, often kept in a shared catalogue called a registry. Each feature has a name, a key such as applicant id, a time window, an owner and a version.
• Pipelines. Jobs that read raw data and compute feature values from those definitions, either on a schedule or as events arrive.
• An offline store. A large, cheap store that keeps every value a feature ever had, each with a timestamp, for building training data.
• An online store. A small, fast key-value store that keeps only the latest value per key, for lookups during a live request.

Uber described a well-known early example in September 2017, in a post about Michelangelo, its machine learning platform. Its central feature store held about 10,000 features at the time. Teams could reuse features other teams had built.

Training-serving skew is any difference between the data a model learned from and the data it sees when serving live requests. The model then decides on inputs it never really learned from.

A 2019 paper on data validation for machine learning, by engineers who ran large production ML systems, splits this skew into three kinds. The first is feature skew: one feature takes different values in training and at serving time.

Their simple example is close to the loan app's problem. A developer adds or removes a feature in the training code path, but forgets to do the same in the serving path.

The second kind is distribution skew: the mix of values in training data differs from serving, for example because of a flawed sampling scheme. The third is scoring and serving skew. A model scores 100 videos but shows only 10, so the 90 unshown videos may never get labels for training.

The same paper names a sneaky cause of feature skew: time travel. A training feature is read from a data source that kept changing after the event. Their example is an ad's click count, which keeps growing after the ad was shown.

The loan app's missed EMI count is exactly this. When the borrower applied, they had missed 1 EMI. By the time the notebook query ran, the table showed 3. The model learned that 3 missed EMIs predicts default, which is true but useless at application time.

In the diagram below, follow the two paths into the model. The same feature names are computed in two separate boxes.

This is not a rare mistake by careless people. A well-known 2015 paper on hidden technical debt in ML systems warns about glue code: code that moves data in and out of general-purpose ML packages.

It says a mature system built that way might end up at most 5% machine learning code and at least 95% glue code. It also describes pipeline jungles: data preparation that grows one scrape and join at a time, until it is costly to test.

The offline store keeps the full history of every feature for training, and the online store keeps only the latest value of each feature for live lookups. Both are filled from the same definitions.

They exist because training and serving want opposite things. Training reads millions of rows at once and can wait minutes. Serving reads one applicant's 40 features and must answer in a few milliseconds.

Uber's Michelangelo used this split. Offline data lived in an HDFS data lake, a file system spread across many machines, queried with Spark and Hive, two big-data processing tools. Features for online predictions were precomputed and stored in Cassandra, to be read at low latency.

Latency means how long one request takes to come back. The table compares the two halves for the loan app.

Feature stores fill the two halves in one of two ways. In some, the job that computes a value writes it to the online store and logs it to the offline store, as Uber's near-real-time pipeline did. Others, such as Feast, load values from the offline data into the online store in a separate step called materialisation. Feast can also take values pushed straight into the online store.

Either way, both copies start from one computation. They can still drift apart if one write fails or a backfill, a job that recomputes old values, rewrites history. That is why a later section adds a skew check.

The next diagram shows the loan app's new setup. Its streaming job reads UPI credits from Kafka, an event streaming system, as they happen. Follow the arrows out of the two jobs into both stores.

Here is the core idea in plain Python, with a list standing in for the offline store and a dictionary for the online store. Look for the one function both sides share.

from datetime import date, timedelta

def avg_monthly_inflow(credits, as_of):
    """credits: list of (day, rupees). The one shared definition."""
    start = as_of - timedelta(days=90)
    window = [amount for day, amount in credits if start <= day < as_of]
    return round(sum(window) / 3, 2)

offline = []  # history: every value with the day it was computed
online = {}   # latest value per (applicant, feature)

def nightly_job(applicants, today):
    for applicant_id, credits in applicants.items():
        value = avg_monthly_inflow(credits, as_of=today)
        offline.append((applicant_id, "avg_monthly_inflow", value, today.isoformat()))
        online[(applicant_id, "avg_monthly_inflow")] = (value, today.isoformat())

credits = {101: [(date(2026, 7, 10), 30000), (date(2026, 8, 10), 30000),
                 (date(2026, 9, 10), 30000), (date(2026, 10, 10), 45000)]}
nightly_job(credits, today=date(2026, 10, 1))
nightly_job(credits, today=date(2026, 10, 19))
for row in offline:
    print("offline:", row)
print("online:", online[(101, "avg_monthly_inflow")])

We ran it on Python 3.13 and got exactly this:

offline: (101, 'avg_monthly_inflow', 30000.0, '2026-10-01')
offline: (101, 'avg_monthly_inflow', 35000.0, '2026-10-19')
online: (35000.0, '2026-10-19')

What just happened? The window line fixes 90 days and stops just before as_of, so no later credit can sneak in. The offline list kept both nightly values. The online dictionary kept only the latest, 35,000 rupees, which is all a live request needs.

A point-in-time join builds each training row from feature values as they were at that row's own timestamp, never later. It stops data leakage from the future, which is when facts from after the event slip into training data. Leakage in general is any information in training that the model will not have live.

Back to the akhada. The selectors keep a register of each wrestler's weight, with the date it was taken. To judge whether a wrestler belonged in a class in March, they read the March entry, never the June one.

The offline store is that register. Every feature value carries the time it became true. For each past application, the join asks one question: what was the latest value known before this moment?

1. 1

   Take one past application

   Applicant 101 applied on 5 March 2026 and did not default. That date is the row's timestamp.

2. 2

   List the stored values

   The offline store holds three snapshots of missed EMI count for 101: 0 on 31 January, 1 on 30 April, and 3 on 30 June.

3. 3

   Keep only earlier values

   Only the 31 January value existed before 5 March. The April and June values are in the future for this row, so they are ignored.

4. 4

   Pick the latest of those

   The training row gets missed EMI count 0, exactly what the live API would have seen on 5 March.

In Python, the pandas library has a function built for this, called merge_asof. Its default direction, backward, picks the last row whose time is less than or equal to the left row's time.

Pause and predict. The same applicant applied again on 20 June 2026 and then defaulted. Which missed EMI count should that row get: 0, 1 or 3?

import pandas as pd

applications = pd.DataFrame({
    "user_id": [101, 101],
    "ts": pd.to_datetime(["2026-03-05", "2026-06-20"]),
    "defaulted": [0, 1],
})
missed_emis = pd.DataFrame({
    "user_id": [101, 101, 101],
    "ts": pd.to_datetime(["2026-01-31", "2026-04-30", "2026-06-30"]),
    "missed_emi_count": [0, 1, 3],
})

rows = pd.merge_asof(applications.sort_values("ts"), missed_emis.sort_values("ts"),
                     on="ts", by="user_id", direction="backward",
                     allow_exact_matches=False)
print(rows[["ts", "missed_emi_count", "defaulted"]])

On pandas 3.0.6, this prints:

ts  missed_emi_count  defaulted
0 2026-03-05                 0          0
1 2026-06-20                 1          1

What just happened? The June row got 1, the 30 April value, because the 3 was stored on 30 June, after the application. A plain join on the latest value gives 3 to both rows. That 3 is the time travel the notebook did.

The by argument matches rows for the same user first. Both tables must be sorted by time, though not by user. When we left the applications unsorted, pandas raised ValueError: left keys must be sorted. Setting allow_exact_matches to False skips a value stamped at the very same moment, a safe choice when you cannot tell which came first.

Real feature stores run this join for you, across dozens of features at once. The idea stays the same: one timestamp per training row, and a backward search for each feature. If you are new to this, you can stop here with the whole core idea.

It computes features ahead of time and keeps the latest values in a fast key-value store. A live request then does one lookup by key instead of running heavy queries.

A key-value store saves each value under a key and fetches it by that key, much like a dictionary in Python. For the loan app, the key is the applicant id and the value is that applicant's 40 latest features.

Computing average monthly inflow live would mean scanning 90 days of bank credits while the buyer waits. On our test machine, a round trip to Redis averaged between about 0.2 and 1.7 milliseconds across runs, with client and server on the same machine.

You can measure that round trip yourself. redis-cli, the Redis command-line tool, sends a PING, a tiny do-nothing request, 100 times a second and reports the fastest, slowest and average reply times.

redis-cli -h ONLINE_STORE_HOST -p 6379 --latency

It keeps updating one line, in milliseconds, until you press Ctrl+C. We got this on Redis 7.0.15 with the client and server on one machine, so there was no network hop. Run it from your API's own server for an honest number.

min: 0, max: 2, avg: 0.32 (473 samples)

The diagram shows one loan request. Each box is one step, and its note says which part talks to which. Notice the branch for missing features.

Two choices in that path need explaining. The first is the missing branch. A brand new applicant may have no stored features yet, so the API sends that request to a person rather than letting the model guess from blanks.

The second is logging. The API saves the exact features each decision used, with the request time. Later, those logged values can be compared with what the offline store says for the same moment. Any gap is skew, caught early.

Features need to be as fresh as the decision demands. Slow-changing features can refresh once a day, while fast-changing ones that drive risky decisions need updates within seconds or minutes.

Freshness is the time between something happening in the real world and the feature showing it. Fresher features cost more, because streaming jobs run all day and need more care than a nightly job.

Uber's 2017 post gives a clean pair of examples from its food delivery app. A restaurant's average meal preparation time over the last seven days was precomputed in batch. The same average over the last hour needed near-real-time compute.

For that, metrics were published to Kafka, and jobs on Samza, a stream processing tool, computed the features. Results were written directly to Cassandra for serving and logged back to HDFS for future training.

The loan app sorts its 40 features by how fast they must change.

• Daily: missed EMI count, past loans closed, average monthly inflow. These change slowly, so the nightly job is enough.
• Within a minute: UPI credits in the last 24 hours, and loan applications from this phone in the last hour. A burst of applications can mean fraud.
• Never stored: the loan amount and showroom in this request. These arrive with the request itself and go straight to the model.

That last group are called request-time features. They still need one shared definition, or the same cleaning code, in both training and serving. The city mismatch was this kind of bug: one field cleaned two ways.

Every online value should also carry the time it was computed. The loan app sets a maximum age per feature. A daily feature older than 36 hours counts as stale, and the request goes to review.

Four details catch out experienced engineers: timestamp ties, rows that vanish, backfills and stale online reads. We ran the pandas checks on pandas 3.0.6 and the Feast checks on Feast 0.67.0, the latest releases on 19 October 2026. Feast 0.67.0 needs pandas below 3, so its runs used pandas 2.3.3.

Ties at the same timestamp

By default, merge_asof uses a value stamped at exactly the application's time. An application on 30 April got missed EMI count 1, the 30 April value. With allow_exact_matches=False it got 0 instead.

Feast's file offline store also used the same-moment value in our run. Neither answer is wrong. What matters is whether the value was really known when the decision was made, so decide the rule on purpose.

Rows with no usable value can vanish

Feast lets each group of features, called a feature view, set a ttl (time to live): how far back the join may look. The Feast docs say it scans backward from each row's timestamp up to that ttl.

The docs say a row with nothing inside its ttl cannot be joined. They do not say it leaves the result. With Feast 0.67.0's default file offline store and a 40-day ttl, we sent in 4 rows and got 3 back.

The missing row was the 20 June application, whose newest earlier value was 51 days old. It was the application where the borrower defaulted, the row a risk model most needs. A user id with no stored values at all was kept, with an empty value.

The source explains it. That store joins each application to every stored value for its key, then removes values outside the ttl window. When none survive, the application row goes too. Even with no ttl, an application dated before its key's first stored value vanished the same way.

This is undocumented as of 19 October 2026, and we tested only the file offline store, so pin your version. The cheap guard is to compare the number of rows you send in with the number you get back.

pandas behaves differently. Its tolerance argument also sets a maximum age, but it keeps the row and leaves the value empty, shown as NaN. With a 30-day tolerance, both example applications kept their rows with empty counts.

Backfills that rewrite history

A backfill can write a corrected value today with an old event time. By default, Feast limits only the event time, so the later correction can reach training rows dated before it existed.

In our run, a 30 April value corrected on 15 July reached a 20 June application. Passing filter_by_created_timestamp=True brought back the original value, because the join then also checks when each value was written. This needs a created-time column on the source; without one, nothing changes. The Feast docs describe this flag, and warn that offline stores without support for it raise an error.

Stale online reads

The ttl did not protect online reads in our run. Using Feast's SQLite online store, a small database kept in one file, a value stamped 30 June came back over 100 days later, though the ttl was 40 days. Feast marked it PRESENT, its status for a found value.

The ttl does change loading. Feast's first incremental materialisation loads only values from the last ttl, as its code documentation says. So read each value's timestamp and enforce your own maximum age, as the loan app does.

Log the exact features each live decision used, rebuild them from the offline store for the same keys and times, and compare them one feature at a time.

The 2019 data validation paper describes the same move. Its validator joins training and serving data on a key, then compares the two feature by feature.

Here is the comparison step in plain Python. The served list is what the API logged. The rebuilt dictionary is what the offline store says for the same requests. Floats are compared with a small tolerance, because tiny rounding gaps are not skew.

import math

served = [  # logged by the live API: (applicant, feature, value)
    (101, "avg_monthly_inflow", 35000.0),
    (102, "avg_monthly_inflow", 12000.0),
    (103, "city", "AHMEDABAD"),
]
rebuilt = {  # recomputed from the offline store for the same request times
    (101, "avg_monthly_inflow"): 35000.0,
    (102, "avg_monthly_inflow"): 18500.0,
    (103, "city"): "Ahmedabad",
}

def same(a, b):
    if isinstance(a, float) and isinstance(b, float):
        return math.isclose(a, b, rel_tol=1e-9)
    return a == b

gaps = [(a, f, v, rebuilt[(a, f)]) for a, f, v in served if not same(v, rebuilt[(a, f)])]
print(f"{len(gaps)} of {len(served)} served values differ")
for gap in gaps:
    print(gap)

Run it and you should see:

2 of 3 served values differ
(102, 'avg_monthly_inflow', 12000.0, 18500.0)
(103, 'city', 'AHMEDABAD', 'Ahmedabad')

What just happened? Two of three values differ, and each gap names the applicant and the feature. The inflow gap points to a different window or a late credit. The city gap points to cleaning code that differs between paths.

You often do not need a feature store when you have one or two models, one team, and predictions made in batch rather than live. A shared Python library of feature functions can be enough.

A feature store adds moving parts: two stores, jobs to watch, a registry to keep tidy. Someone has to own all of that. For a small team, that cost can outweigh the bugs it prevents.

These signs say you can wait:

• Predictions run in a nightly batch, so training and scoring can share one script.
• Only one or two models use the features, owned by the same people.
• No feature needs to be fresher than a day.
• You can cover skew with one feature library imported by both paths, plus the nightly comparison above.

These signs say a feature store will pay for itself:

• Live predictions need features within milliseconds.
• Several teams rebuild the same features, like inflow or repayment history, in their own ways.
• You have already shipped a skew or leakage bug, as the loan app did.
• You need to show auditors exactly which values drove a past decision.

That last point counts at a lender. If a customer asks why their loan was refused, the served-feature log shows the exact values the model used, with the time of the decision.

Explain it as the answer to training-serving skew, then walk through the two stores, the point-in-time join and freshness, using one concrete model as your example.

1. 1

   Start with the model and its features

   Say what the model predicts and list four or five features, with their keys and windows. A loan risk model is a good, simple example.

2. 2

   Show the skew problem

   Draw training and serving computing features separately. Give one concrete mismatch, like a 90-day window in training and a 30-day one live.

3. 3

   Add one set of definitions

   Introduce the registry and the jobs that compute every feature from it, so no feature is written twice.

4. 4

   Split offline and online

   Explain why history and latest values live in different stores, and name realistic choices such as object storage and Redis.

5. 5

   Cover time correctly

   Explain point-in-time joins for training and a maximum age for online values. Say what happens when a feature is missing or too old.

6. 6

   Close with checks

   Log served features, compare them with offline values each night, and count training rows in and out of every join.