Class Imbalance: When Fraud Is 1 in 1,000

It can catch no fraud at all. When fraud is 1 payment in 1,000, a model that calls every payment legitimate is right 999 times out of 1,000. This is class imbalance, and accuracy rewards the model for ignoring the one case you care about.

Picture a common setup. A payment gateway screens about 10 lakh (1 million) online payments a day for fraud. Roughly 1 in 1,000 turns out to be fraud, so about 1,000 bad payments a day. The setup is made up, but the problem is real.

Each fraud costs the gateway about Rs 4,000 on average, mostly through chargebacks. A chargeback is when a card holder disputes a payment and their bank takes the money back. The team often learns a payment was fraud only when the chargeback lands, weeks later.

The team trains its first fraud model on three months of past payments, and the report says 99.9% accuracy. Then the head of risk operations asks how many of yesterday's frauds it stopped. The answer is zero, because the model learned to approve everything.

Nothing in the code was broken. The data was lopsided, and the score the team chose hid that. This guide fixes the score first, then the training, then the decision rule, all on this one gateway.

Class imbalance is when one label in your data is much rarer than the others. In fraud screening, legitimate payments outnumber fraud by hundreds or thousands to one.

A class is one answer the model can give, here fraud or legitimate. The rare class is called the minority class. The common one is the majority class.

This is normal, not an edge case. A public research dataset of European card payments from September 2013 holds 284,807 transactions over two days. Only 492 of them are fraud, about 0.17%.

Picture a silk farmer in his rearing shed, with trays holding thousands of cocoons. Perhaps one cocoon in a thousand is diseased. A helper who glances at each tray and says all healthy is almost always right.

That helper is also useless. The one sick cocoon is the whole reason anyone checks the trays. Left alone, it can spread disease and spoil a season of silk.

Your model feels a similar pull. Training rewards it for fitting the examples it sees, and 999 in 1,000 of them say legitimate. Leaning hard towards that answer is the cheapest way to look good.

Imbalance is not always the real problem

A lopsided ratio alone need not ruin a model. A 2022 study of risk models built with logistic regression, a standard model that outputs probabilities, tested random undersampling, random oversampling and SMOTE, three common corrections explained below. None helped the models rank cases better, and all made rare-class risks far too high.

In that study, the ratio itself was not what hurt the models. So check the other suspects before blaming it: a wrong metric, a wrong threshold, or too few rare examples. The rest of this guide deals with each one in turn.

Use metrics that focus on the rare class: recall, precision and the precision-recall curve. Read them from a confusion matrix, not from a single accuracy number.

A confusion matrix is a small table that counts four outcomes. Each payment is either fraud or not, and the model either flags it or not. That gives four boxes.

• True positive: a fraud the model flagged. This is a catch.
• False positive: a legitimate payment the model flagged. This is a false alarm that annoys a real customer.
• False negative: a fraud the model let through. This is a miss, costing about Rs 4,000.
• True negative: a legitimate payment the model let through. This is the quiet majority.

Recall is the share of real frauds the model catches: true positives divided by all frauds. Precision is the share of flags that really are fraud: true positives divided by all flags.

In the silk shed, recall asks how many sick cocoons the farmer pulled out. Precision asks how many of the pulled cocoons were really sick, rather than good silk set aside for nothing.

Suppose the team's second model flags 3,000 payments a day, and 700 of them are real fraud. The table compares it with the first, lazy model on the same day.

Look at the first row. The far better model has lower accuracy. Its 2,300 false alarms cost it more accuracy points than its 700 catches earn back.

ROC curve or precision-recall curve?

Most models give each payment a score, and you choose a cut-off. A curve shows the results at every cut-off at once. The area under it sums that up as one number.

The ROC curve (short for receiver operating characteristic) plots recall against the false positive rate, the share of legitimate payments wrongly flagged. With 999,000 legitimate payments, 2,300 false alarms is a rate of only 0.23%. So on a ROC chart the model looks excellent: 70% recall at a false positive rate near zero.

The precision-recall curve plots precision against recall instead. Precision feels every false alarm, because it sets them beside the few real frauds. That is exactly the comparison the review team lives with.

A 2015 study by Saito and Rehmsmeier tested this with simulations and real data. It showed that ROC plots can be deceptive on imbalanced data, and that precision-recall plots give a more accurate picture. Their baseline moves with the class ratio.

That baseline is worth remembering. A model that guesses at random gets a ROC area of about 0.5 on any data. On the precision-recall curve, the same guesser's precision is just the fraud rate, 0.001 here.

Check the accuracy trap yourself

If you code in Python, this short script proves it; if not, skip to the result below it. It builds 10 lakh labels with exactly 1,000 frauds. It scores a model that always says legitimate, then a model that guesses at random.

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score
from sklearn.metrics import roc_auc_score, average_precision_score

rng = np.random.default_rng(0)
y = np.zeros(1_000_000, dtype=int)
y[rng.choice(y.size, 1_000, replace=False)] = 1   # 1 fraud in 1,000
X = np.zeros((y.size, 1))                          # features do not matter here

lazy = DummyClassifier(strategy="most_frequent").fit(X, y)
pred = lazy.predict(X)
print("accuracy", accuracy_score(y, pred))
print("recall", recall_score(y, pred))

guess = rng.random(y.size)                         # a model that guesses at random
print("ROC area of guessing", round(roc_auc_score(y, guess), 2))
print("PR area of guessing", round(average_precision_score(y, guess), 3))

On scikit-learn 1.9.1, the latest release as of 19 October 2026, it prints accuracy 0.999 and recall 0.0. The random guesser gets a ROC area of 0.51 and a PR area of 0.001.

The PR area here is average precision, scikit-learn's summary of the precision-recall curve. A real model has to beat 0.001 on it, not 0.5, before it has learned anything.

Use sampling only on training data, and keep it only if it wins on an honest test set. For fraud, undersampling is usually the cheapest first try, because legitimate payments are plentiful.

Sampling changes the mix of examples the model learns from. It never changes the real world the model will face. There are three common kinds.

• Random undersampling keeps only some majority examples, such as 1 legitimate payment in 100. Training gets faster and cheaper, but you throw information away.
• Random oversampling copies rare examples several times. Nothing is thrown away, but the model may memorise those exact payments.
• SMOTE creates new, synthetic rare examples that sit between real ones. Chawla and colleagues published it in 2002 in the Journal of Artificial Intelligence Research.

SMOTE stands for synthetic minority over-sampling technique. It takes one fraud example and one of its nearest fraud neighbours, from the 5 nearest by default in the imbalanced-learn library. Then it creates a new point somewhere on the line between them.

The original paper paired SMOTE with undersampling the majority class. That combination did better than undersampling alone in their tests, measured with ROC curves on several public datasets.

SMOTE has a catch for payments. Real features mix numbers with categories, like merchant type. A point halfway between a grocery payment and a jewellery payment may describe no real payment at all.

imbalanced-learn ships a variant, SMOTENC, for exactly this mix. In version 0.14.2 it gives each new point the category most common among its neighbours, so merchant type stays a real merchant type.

Split first, sample second

Always split your data before you sample anything. If copies or synthetic neighbours of test frauds leak into training, the model has already seen the answers. Your scores will then be far too good.

Fraud data also changes over time, so split by date, not at random. Train on older weeks and test on newer ones. That copies real life, where the model always faces next week's fraud.

Here is the order to follow. Notice that sampling sits on one branch only. The validation and test weeks never pass through it, so they keep the real 1 in 1,000 mix.

Class weights tell training that a mistake on a fraud counts more than a mistake on a legitimate payment. You keep every row and change only how much each one matters.

During training, the model adds up a loss, a number that grows with every wrong answer. It then adjusts itself to make that number smaller. Weights change how much each wrong answer adds.

A common default is called balanced weights. Each class gets total rows divided by two times that class's rows. Both classes then add the same total weight to the loss.

For this gateway, fraud gets a weight of about 500 and legitimate payments about 0.5. Missing one fraud now hurts as much as about a thousand mistakes on legitimate payments.

Neither fix always wins. Weights keep all the data and suit any model that accepts them. Undersampling makes training on billions of rows much cheaper. Try both on the same test weeks and keep the winner.

Both make fraud look more common in training than it really is. The model learns that higher base rate, the share of fraud it expects before looking at any payment. So its probabilities come out too high, even when it still ranks payments well.

A probability is calibrated when it means what it says. Of all payments scored 0.2, about 20% really are fraud. In the 2022 study, undersampling, oversampling and SMOTE all pushed rare-class probabilities strongly too high.

The formula for random undersampling

Keep 1 legitimate payment in 100 and every fraud. The training mix is now 1 fraud for every 10 legitimate payments, not 1 for every 999. The odds of fraud look 100 times higher than they are.

Bayes' rule, the rule for updating a probability when the base rate changes, gives a formula that divides that back out. Dal Pozzolo and colleagues set it out in a 2015 paper on calibrating probabilities after undersampling.

def correct_probability(p_sampled, beta):
    """Map a score from an undersampled model back to the real fraud rate.

    beta is the share of legitimate payments kept in training.
    Keeping 1 in 100 means beta = 0.01.
    """
    return beta * p_sampled / (beta * p_sampled - p_sampled + 1)


print(round(correct_probability(0.5, 0.01), 4))   # 0.0099
print(round(correct_probability(0.9, 0.01), 4))   # 0.0826

The two printed results are the lesson. A payment the trained model scores 0.5 is really about a 1% fraud risk. Even a score of 0.9 means only about 8%.

The formula is exact only if the model's scores were right for the sampled mix. A 2024 paper by Phelps, Lizotte and Woolford points out that it does not correct errors in the model itself.

In our simulation on synthetic data with scikit-learn 1.9.1, a logistic regression trained with 1 in 100 legitimate rows kept gave scores averaging 0.084. After the correction it averaged 0.00169, against a real fraud rate of 0.0017.

The correction never changes which payment scores higher than which. It only fixes what the numbers mean. That matters when you read a score as a real risk, for example to check that a review is worth its cost.

What about class weights?

Weights have the same side effect. In our test, a logistic regression with balanced weights averaged a fraud score of 0.23 on data where the real fraud rate was about 0.0017.

The same formula, with beta set to the legitimate weight divided by the fraud weight, brought that average down to 0.0019. That is close but not exact, so treat it as a first step, not a proof.

So with either fix you have two honest choices. Fit a calibration step on the real-rate validation week, which maps scores to true fraud rates. Or treat scores purely as a ranking and choose the cut-off on that week.

Choose the threshold that gives the lowest total cost on a real-rate validation set, within what your review team can handle. The usual default of 0.5 means nothing special when fraud is rare.

A threshold is the cut-off score above which you act on a payment. Here, acting means holding the payment for the risk team's analysts to check by hand.

Each choice has a price. A missed fraud costs about Rs 4,000. A manual review costs about Rs 40 of analyst time. The team can review at most 3,000 payments a day.

Back in the silk shed, the farmer cannot cut open every cocoon. He sets aside the ones that look slightly off for a closer look. Too strict, and his whole day goes on healthy cocoons. Too loose, and disease spreads through the trays.

Where the picture breaks: the farmer can look at a whole cocoon, while a model sees only the few features you give it. And diseased cocoons do not change their look to slip past him, but fraudsters keep changing tactics.

The helpers below turn this into code. They take scores and true labels for one held-out day at the real fraud rate, as NumPy arrays. The search only counts flags and misses, so raw scores work too. It tries every cut-off from 0.001 to 0.999.

import numpy as np

FRAUD_LOSS = 4000     # rupees lost per missed fraud (made up)
REVIEW_COST = 40      # rupees per manual review (made up)
CAPACITY = 3000       # reviews the team can do in a day (made up)

def day_cost(scores, labels, threshold):
    flagged = scores >= threshold
    missed = int(np.sum(labels[~flagged]))
    reviews = int(np.sum(flagged))
    return missed * FRAUD_LOSS + reviews * REVIEW_COST, reviews

def pick_threshold(scores, labels):
    best = None
    for t in np.arange(1, 1000) / 1000:
        cost, reviews = day_cost(scores, labels, t)
        if reviews <= CAPACITY and (best is None or cost < best[0]):
            best = (cost, float(t))
    return best   # None means no cut-off fits the team's capacity

Two lines matter most. The return line of day_cost puts misses and reviews on one scale, rupees. The capacity check rejects any cut-off that would bury the analysts, however cheap it looks.

In our test on 10 lakh synthetic rows, the full search took about 22 seconds on one CPU core; your time will differ. It returns a pair: the lowest daily cost in rupees and the cut-off that gives it.

On the validation day (made up), the best cut-off flags 3,000 payments and catches 700 frauds. The daily cost falls from Rs 40 lakh with the lazy model to Rs 13.2 lakh.

Corrected scores give a quick check on this. If a review stops the fraud it finds, it pays when risk times Rs 4,000 beats Rs 40, so above a risk of 0.01. With the cap removed, our search on synthetic calibrated scores picked a cut-off of 0.012. In this scenario more than 3,000 payments clear 0.01, so the cap decides.

If you use scikit-learn, versions 1.5 and later include TunedThresholdClassifierCV. It picks a cut-off by cross-validation, refitting on several splits of the training data, to maximise a score you choose, balanced accuracy by default. It has no review-capacity setting, so encode that cap in a custom scorer or keep a loop like this.

The team later adds a second, higher cut-off. Payments above it are blocked at once, because they are almost always fraud. Only the band between the two cut-offs goes to review.

It keeps the real fraud rate everywhere except training, acts through tuned cut-offs, and feeds analyst verdicts and chargebacks back as new labels. The full fraud detection system design is a topic of its own.

Follow one payment from the top of the diagram. Notice the two places labels come from. One is fast but covers only flagged payments, and one is slow but covers everything.

Two details here catch beginners out. The first is label delay. A fraud is often confirmed only when its chargeback lands, which can take weeks.

If the team trains on last week's payments, many frauds still look legitimate. The fraud rate then looks even lower than it is. So it trains only on payments old enough for their labels to settle.

The second is lopsided feedback. Analysts only check what the model flagged. Approved payments get labels only through chargebacks, and some fraud is never reported at all.

So the team sends a small random sample of approved payments to review each week. It shows where the model is blind. Choosing which unlabelled cases people should check is the core idea of active learning.

The most common mistakes are sampling before splitting, splitting at random, and trusting accuracy or raw probabilities. Each one gives you numbers that will not hold in production.

• Sampling before splitting. Copies of test frauds leak into training, so test scores become fiction.
• Random splits on data that changes over time. The model partly learns from later fraud patterns and is tested on earlier ones. Split by date instead.
• Testing on a balanced set. The second model's 23% precision would read 99.7% on a 50 to 50 test set, with the same recall and false alarm rate.
• Too few frauds in the test weeks. With 30 frauds and a true recall of 70%, measured recall lands anywhere from 53% to 87% in 95% of runs. With 300 frauds the range is 65% to 75%.
• Reading sampled probabilities as real. A score of 0.9 from a model trained on a 1 to 10 mix may mean about 8% risk.
• Keeping the 0.5 cut-off. Nobody chose it for your costs or your team's capacity.
• Expecting a trick to fix missing data. With only a few dozen frauds, start with rules and anomaly detection, and collect labels.

Anomaly detection means flagging payments that look unlike normal traffic, without learning from fraud labels. It is a useful start when confirmed frauds are scarce. It cannot tell an unusual honest payment from fraud.

Explain it as a chain: the rare class, the right metric, the training fix, the threshold, and the feedback loop. Tie every step to money and to the people who act on flags.

1. 1

   State the ratio and the volume

   Ask for the fraud rate and daily traffic. At 1 in 1,000 on 10 lakh payments, you expect about 1,000 frauds a day.

2. 2

   Retire accuracy early

   Point out that approving everything scores 99.9%. Propose recall, precision and the precision-recall curve on the fraud class instead.

3. 3

   Split by time

   Train on older weeks. Validate and test on newer weeks that keep the real fraud rate.

4. 4

   Fix the training

   Offer class weights or undersampling of legitimate payments. Say you will correct or recalibrate the probabilities afterwards.

5. 5

   Set cut-offs from costs

   Compare the cost of a miss with the cost of a review. Respect the review team's daily capacity, and add a block cut-off for clear cases.

6. 6

   Close the loop

   Describe label delay, random audits of approved payments, and scheduled retraining with a freshly tuned threshold.