ML Model Monitoring: Data Drift and Concept Drift

The model keeps answering quickly and confidently while its answers get worse. Nothing crashes, so the server dashboard stays green and the damage shows up later in business numbers. This is model drift, and it happens because the world changed while the model stayed frozen.

Picture a common setup. A microfinance lender in Madurai gives small loans to women's self-help groups, which are groups of 10 to 20 women who save together and borrow as a group. The setup is made up, but the problem is real.

The lender's risk model scores each loan application from 0 to 100. A high score means the group is likely to miss repayments. The data team trained it on 18,000 past loans.

For its first year, it worked well. Groups the model called low risk missed payments about 2 times in 100. Loan officers trusted the score and approved loans in minutes instead of days.

In the second year, two things changed that the model had never seen. Neither one touched the code.

• The lender opened branches in rural villages. The new groups asked for smaller loans and had shorter savings records than most groups in the training data.
• A long dry spell hit jasmine and cotton growers in the area. Groups that looked like good borrowers on paper started missing their weekly payments.

Within months, low-risk groups were missing payments 7 times in 100. The scoring service still answered every request in 80 milliseconds (made up) and never threw an error. The code was fine; the world had moved.

This guide follows the same lender from the first warning sign to a new model in production. At each step it shows how to check the result yourself.

You need four words: feature, label, distribution and label delay. Each one gets a plain meaning and an example from the lender.

Feature. A feature is one input the model reads to make its guess. Loan amount, months of group savings and the number of members are three features of an application.

Label. A label is the true outcome the model is trying to predict. For each past loan, the label is yes if the group fell 30 days behind on payments, and no otherwise.

Distribution. A distribution is the overall shape of a set of values: which values are common and which are rare. In the training data, about 8 in 10 loans were between Rs 25,000 and Rs 63,000.

Label delay. Label delay is the wait between making a prediction and learning the true outcome. The lender's loans run for 12 months, so a loan approved today is fully judged only a year later.

Model drift is the drop in a model's quality after deployment, when live inputs, or the link from inputs to outcomes, move away from what it learned in training. People also call it model decay.

A model learns one fixed pattern: how inputs and outcomes go together in its training data. Researchers write the inputs as X and the outcome as y. A 2018 review by Lu and colleagues defines concept drift, in its broad sense, as a change over time in this joint pattern. Joint means how often each mix of inputs and outcome shows up.

That joint pattern can change in two places. The inputs alone can change, which the review calls virtual drift and monitoring tools call data drift. Or the link from inputs to outcome can change, which the review calls actual drift and tools call concept drift.

The review also notes that in many real-world applications both happen together. The lender had exactly that.

Think of Makar Sankranti, when rooftops fill with kite flyers. A flyer spends years learning to steer in the usual gentle afternoon wind. That practice is the training data, and the flyer's moves are the model.

Now the wind shifts. It blows harder and from the east, but the rules of the kite fight stay the same. The old moves were learned for other conditions, so the kite wobbles. That is data drift: the inputs changed.

Next year the wind is normal, but the rules change. Cutting other strings no longer wins; the highest kite wins. The old moves now lose in perfect weather. That is concept drift: what counts as right has changed.

Where the picture breaks: a flyer feels the new wind at once and adjusts. A model never notices anything. It keeps flying the same way until someone measures the results.

The diagram below shows where each kind of drift enters the lender's system. Follow the arrows from the real world to the final repayment outcome.

Data drift is a change in the inputs a model sees, while concept drift is a change in what those inputs mean for the outcome. With data drift the rules still hold; with concept drift the rules themselves moved.

A 2014 survey by Gama and colleagues describes concept drift in these terms: the relation between the input data and the target variable changes over time. The target is the thing being predicted, here a missed repayment.

The lender had both at once. The village branches were data drift: smaller loans and shorter savings records that the model had rarely seen. The dry spell was concept drift: a group with three years of savings used to be safe, and now it was not.

Data drift can still hurt accuracy

In theory, pure data drift leaves the right answer for each input unchanged. Lu's review says it does not move the decision boundary, the true line between risky and safe. But your model is only an estimate of that line, and the estimate is weak where training data was thin.

We tested this on made-up loans with savings length as the only feature, using scikit-learn 1.9.1. The training loans had 12 to 60 months of savings. When live groups had 30 to 60 months, inside that range, the model expected 3.1 misses in 100 and the real number was 2.8.

When live groups had only 1 to 12 months, below the training range, the model expected 5.7 misses in 100 and the real number was 14.5. The true rule never changed. The model had simply never learned that corner.

There is a third thing that looks like drift but is not. A pipeline bug is a mistake in the code that prepares the data, such as a field sent in rupees one day and in thousands of rupees the next. It is a common false alarm, so it gets its own column below.

Drift also differs in how it unfolds over time. Lu's review names four shapes, and each can appear in lending.

• Sudden: a new pattern arrives within a short time, like a new government rule on loan interest.
• Gradual: a new pattern slowly replaces the old one over a period of time.
• Incremental: the old pattern itself changes step by step, like rising prices that slowly shrink what a loan can buy.
• Recurring: an old pattern comes back after some time, like repayment trouble every year before harvest. The review calls this reoccurring drift.

Beginners can stop here. You know what drift is, its two kinds, and the false alarm that imitates them. The rest of the guide is for engineers who build the checks.

You detect data drift by comparing each live feature against a reference window, a fixed past sample such as the training data, and alerting when the gap is large. You need no labels, so you can run it every day.

The basic tool is a two-sample test: a statistics check that asks whether two sets of values could come from the same distribution. For number features, a common choice is the two-sample Kolmogorov-Smirnov test, or KS test.

For every value, the KS test asks what share of each set falls at or below it. The biggest gap between those two shares is the statistic, from 0 for identical sets to 1 for sets that never overlap. It also returns a p-value: the chance of a gap at least this big if both sets came from the same distribution. A small p-value is evidence that the two distributions differ.

Here is the lender's daily check, using SciPy's ks_2samp function. It runs as it is. The first part makes made-up data: 18,000 training loans and 900 village applications from one week.

import numpy as np
from scipy.stats import ks_2samp

rng = np.random.default_rng(7)
# Made-up data: 18,000 training loans and one week of 900 village applications
train = {
    "loan_amount": rng.lognormal(10.6, 0.35, 18_000),
    "savings_months": rng.gamma(6.0, 5.0, 18_000),
    "members": rng.integers(10, 21, 18_000),
}
live = {
    "loan_amount": rng.lognormal(10.35, 0.35, 900),
    "savings_months": rng.gamma(4.5, 5.0, 900),
    "members": rng.integers(10, 21, 900),
}

def drifted_features(train, live, p_cut=0.01, min_gap=0.1):
    alerts = []
    for name in train:
        result = ks_2samp(train[name], live[name])
        if result.pvalue < p_cut and result.statistic > min_gap:
            alerts.append((name, round(float(result.statistic), 2)))
    return sorted(alerts, key=lambda a: a[1], reverse=True)

print(drifted_features(train, live))

Run it with SciPy 1.18.1 and NumPy 2.5.4, the latest releases as of 19 October 2026. It prints [('loan_amount', 0.28), ('savings_months', 0.26)]. Group size, the members feature, did not move, so it stays quiet.

What just happened? The test compared each feature's live shape with its training shape. Loans got smaller and savings records shorter, so both gaps are large. Nothing has gone wrong with repayments yet, but the model is now scoring people unlike its training data.

The line that matters is the if. It demands both a small p-value and a gap bigger than 0.1. With enough rows, even a tiny, harmless shift gets a tiny p-value, so the gap rule keeps the alert quiet for shifts too small to matter.

In our test with 2 lakh (200,000) rows on each side, we shifted one feature by 0.03 standard deviations. A standard deviation is the typical distance of values from their average. The p-value fell under one in a trillion, but the gap was only 0.012.

So the 0.1 gap is a starting point, not a standard. Tune it on your own history of harmless weeks.

Each feature gets its own test, so more features mean more chances of a false alarm. In our test with 20 unchanged features, p under 0.01 alone fired in 18 of 100 healthy weeks. The 0.1 gap rule stopped all of them with 900 live rows, but none with 100 rows. There, dividing p_cut by the number of features, called a Bonferroni correction, cut it to under 1 in 100.

Population Stability Index, the credit-scoring favourite

Credit teams often use the Population Stability Index, or PSI, instead. It splits the training values into buckets, compares each bucket's share in live data, and adds up the weighted differences. A 2018 thesis by Yurdakul describes it as a widely used model monitoring metric in credit scoring.

def psi(expected_values, actual_values, bins=10):
    edges = np.quantile(expected_values, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected_values, edges)[0] / len(expected_values)
    a = np.histogram(actual_values, edges)[0] / len(actual_values)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

for name in ["loan_amount", "savings_months"]:
    print(name, round(psi(train[name], live[name]), 2))

Run after the first block, it prints loan_amount 0.49 and savings_months 0.37. Common industry rules of thumb read under 0.10 as little shift, 0.10 to 0.25 as moderate, and over 0.25 as significant shift that needs action.

Yurdakul's thesis points out that these cut-offs are used without reference to false alarm or miss rates. They are habits, not statistics. PSI also moves with the bucket count: in our test, loan_amount read 0.47 with 5 buckets and 0.56 with 50, so keep the count fixed.

Other signals worth watching

• Category features, like district or crop type: compare how often each value appears, and watch for brand-new values the model never saw.
• Missing values: a feature that suddenly goes blank for 20 in 100 rows often points to a pipeline bug, not drift.
• The scores themselves: if the share of high-risk scores jumps, something upstream moved, even before you know what.

Why one feature at a time is not enough

Pause and predict. Two features keep exactly the same shape, but the way they move together flips. Will a KS test on each feature notice?

It will not. In our test, two made-up features went from a correlation of 0.8 to minus 0.8, where correlation is a minus 1 to 1 measure of how two values move together. Each feature's KS gap stayed under 0.05, with p-values of 0.07 and 0.12.

The model's own scores caught it. Their KS gap was 0.25, and the share of scores above 0.2 fell from 20 in 100 to about 1 in 100. Watching the score distribution is cheap, and it catches input shifts that change the overall mix of scores. In a second test, it stayed quiet when a feature the model barely used moved, and when only the link from inputs to outcome changed.

This matches the Failing Loudly study by Rabanser and colleagues, published at NeurIPS 2019. It tested image datasets such as MNIST and CIFAR-10, public collections of small pictures of digits and objects. Its best detector first turned each input into the output probabilities of a classifier, a model that sorts inputs into classes, trained on the original data. It then ran a KS test on each of those numbers, with a Bonferroni correction across the tests.

You detect concept drift by tracking real performance on loans whose outcomes are known, and by using earlier proxy labels while you wait for the final ones. Without outcomes, concept drift is mostly invisible.

This is the hard part. The dry spell did not change a single input field, so every input check stayed quiet. Only repayments told the truth, and the lender's label, 30 days behind, takes months to happen.

Lu's review notes that most existing drift detection and adaptation methods assume the true label is available. In lending it arrives late. So the team adds a faster proxy label: did the group miss any of its first eight weekly payments?

A proxy label is an early outcome that tends to agree with the real one. It is less exact, but it arrives in about two months instead of twelve. The model predicts the main label, falling 30 days behind, not the proxy, so its scores are not early-miss rates.

So the check first learns, from a healthy past year, how often loans in each score band, a range of risk scores, missed early. It then compares that expected rate with the actual rate for loans old enough to judge.

from dataclasses import dataclass
import random

@dataclass
class Loan:
    start_month: str
    score_band: int      # risk score band, 0 (safest) to 4
    weeks_seen: int      # weekly payments due so far
    missed_early: bool   # missed any of the first 8 weekly payments

def judged(loans, month):
    return [l for l in loans if l.start_month == month and l.weeks_seen >= 8]

def band_rates(past_loans):
    # early-miss rate per score band, learned from a healthy past period
    seen, missed = {}, {}
    for l in past_loans:
        seen[l.score_band] = seen.get(l.score_band, 0) + 1
        missed[l.score_band] = missed.get(l.score_band, 0) + l.missed_early
    return {band: missed[band] / seen[band] for band in seen}

def concept_check(loans, month, rates, tolerance=1.5, min_loans=500):
    done = judged(loans, month)
    if len(done) < min_loans:
        return "wait", None, None  # too few loans to judge
    expected = sum(rates[l.score_band] for l in done) / len(done)
    actual = sum(l.missed_early for l in done) / len(done)
    status = "alarm" if actual > expected * tolerance else "ok"
    return status, round(expected, 3), round(actual, 3)

# Made-up loans. Healthy past year: band b missed early (b + 1) times in 100.
random.seed(4)
past = []
for _ in range(20_000):
    band = random.randrange(5)
    past.append(Loan("2025", band, 8, random.random() < (band + 1) / 100))
# June, during the dry spell: the same bands miss twice as often.
june = []
for _ in range(1_000):
    band = random.randrange(5)
    june.append(Loan("June", band, 9, random.random() < 2 * (band + 1) / 100))

print(concept_check(june, "June", band_rates(past)))

It prints ('alarm', 0.03, 0.062): June's score bands should have missed early 3 times in 100, and the real rate was about 6. Run in August, that alarm comes about ten months before the full 12-month labels for the June loans are known.

The bands matter because a month of riskier groups should miss more often, and that is not drift. In our test, a fixed baseline of 3 in 100 fired in about half of healthy months made only of the two riskiest bands. The band baseline almost never did.

Two other settings matter. The minimum of 500 loans stops a small, noisy month from raising an alarm. The tolerance of 1.5 pages someone only when misses run 50 percent above the expected rate.

How noisy is a monthly check?

Small months make noisy alarms, and you can work out how noisy. We used the binomial distribution, the maths of counting misses among a fixed number of loans, with SciPy 1.18.1. A healthy month misses early 3 times in 100, as expected, and a broken month 6 times in 100.

So a 200-loan minimum pages someone in about one healthy month in twelve, and still misses about one broken month in four. If your months are small, pool two months for a slower but surer alarm.

The table also assumes loans miss independently, but groups in one village share the same weather. We simulated 25 villages of 20 loans whose miss rates differed from village to village, a made-up spread. The 500-loan false alarm rate rose from 3 to between 6 and 12 per 100, depending on how much villages differed.

Researchers call this family performance-based drift detection. A 2022 overview by Bayram and colleagues reviews detectors that use a drop in the model's own performance to signal a real change. It is the most direct signal, because it measures what you care about.

An ML model monitoring system has two halves: a fast half that watches inputs and scores every day, and a slow half that joins true outcomes to old predictions. Both feed one dashboard and one alert path.

Here is the lender's setup. Start at the loan officer app and follow both paths into the dashboard. The prediction log is the heart of it: without it, you cannot compare anything later.

A few design choices make this work in practice. Each one stops a specific way of fooling yourself.

• Log the exact inputs the model saw, not values looked up again later. The Rules of Machine Learning guide by Martin Zinkevich gives the same advice: save the features used at serving time.
• Log the model version with every score, so a retrain never mixes two models in one chart.
• Keep the reference window fixed and written down, such as the training set, so every gap compares against the same thing.
• Slice every chart by branch, district and loan size. Drift in the new villages can hide inside a healthy-looking average.

Retrain when performance on recent labelled data has really dropped, and only after you have ruled out a pipeline bug. An input shift alone is a reason to look closer, not to retrain.

Lu's review calls retraining on the latest data perhaps the most straightforward reaction to drift. It adds that an explicit drift detector is needed to decide when to retrain. It also names the trade-off in choosing a training window: a small window reflects the newest pattern, while a large one gives more data.

The diagram below is the team's decision path when an alert fires. The pipeline question comes first.

Here is how the team handled the dry spell, from alarm to a new model in production. The first step needed no new model at all.

1. 1

   Stop the bleeding

   They raised the approval cut-off for jasmine and cotton villages and sent borderline cases to a loan officer. This took one day and needed no new model.

2. 2

   Pick the training window

   They trained on the last 9 months, with recent loans weighted more heavily. Old loans still taught the stable patterns, but new ones counted more.

3. 3

   Add what the model lacked

   They added a rainfall feature for each village, because the old model had no way to see the dry spell at all.

4. 4

   Test on the future, not the past

   They trained on loans up to May and tested on June and July. Recent loans have no 12-month label yet, so both sides used the early-payment label. A random split would hide drift, because test rows would share months with training rows.

5. 5

   Run it in shadow

   For two weeks the new model scored every application quietly, beside the old one. Loan officers still saw only the old score.

6. 6

   Switch and keep watching

   They made the new model live, logged its version, and reset the drift checks against its own training window. Because it learned the early-payment label, its scores meant something new, so they also reset the approval cut-off and the expected rate for each score band.

We checked the random-split warning in a simulation of gradual drift with scikit-learn 1.9.1. We scored ranking quality with ROC AUC, a 0 to 1 measure of how often the model ranks a defaulter above a good borrower. A random split overstated that score for the next two months in 9 of 10 runs. The gap was small, about 0.01, but nearly always flattering; a time split's estimate ran about 0.02 low instead, which is the safer error.

Some drift comes back every year, like trouble before harvest. For that, consider a feature that captures the cycle, such as month of year. The model can then learn the cycle from past years instead of being retrained each season.

Classic model drift is measured against a clear true outcome, like a repayment, while apps built on large language models (LLMs) produce free text with no single correct label. So the monitoring tools differ, even though the idea of silent decay is the same.

The lender's model gives one number per loan, and the world eventually says whether it was right. That makes miss rates and KS tests possible. A chat assistant writes paragraphs, and nobody stamps each one right or wrong.

LLM monitoring leans on sampled human reviews, automatic graders, user feedback and cost per request. Our guide on LLM observability, silent regressions and cost drift covers that side. This guide stays with classic models that predict a number or a class.

Explain it as two problems with two clocks: input drift you can see today, and outcome drift you see only when labels arrive. Then show the system that watches both.

1. 1

   Ask about the label

   What is the true outcome, and how long until it is known? A click is known in seconds, a loan default in months. This sets your whole design.

2. 2

   Separate the two drifts

   Define data drift and concept drift in one line each, with an example from the problem you were given. Mention pipeline bugs as the third suspect.

3. 3

   Draw the prediction log

   Every score is stored with its inputs and model version. Say that without it, no later comparison is possible.

4. 4

   Add both checks

   A daily input check with two-sample tests and score shares, and an outcome join that tracks real performance, sliced by segment.

5. 5

   Size the alarms

   Say how many outcomes you need before a monthly check is trustworthy, and what false alarm rate you accept.

6. 6

   Describe the response

   Rule out pipeline bugs, measure impact, apply a quick business fix, then retrain on a recent window and test on the newest data.