Business Metrics vs ML Metrics: Better AUC, Worse Revenue

A better model wins at a job that does not pay. That is the gap behind ML metrics vs business metrics. The model learns to predict something easy to measure, such as a tap, while the business is paid for orders that shoppers keep.

Picture a common setup. An online clothing store in India sells kurtas, jeans, T-shirts and sarees, mostly priced between Rs 199 and Rs 1,499. About 1.2 lakh (120,000) shoppers open its app each day. The setup is made up, but the problem is real.

The home screen has a shelf: a row of eight products picked for each shopper. A model gives every product a score for that shopper, and the eight highest scores fill the shelf.

The ML team trains a new model to predict whether a shopper will tap a product. On last month's saved data, its AUC rises from 0.72 to 0.79. For now, read AUC as a sorting grade where 0.5 is a coin toss and 1.0 is perfect.

The team runs an A/B test for two weeks. An A/B test splits shoppers at random into two groups, and each group sees a different version of the app. Half see the old shelf and half see the new one.

Taps on the shelf go up by 12%. Three weeks after the test ends, once returns are in, the finance report arrives. Net revenue per visitor is the order money the store keeps after refunding returns, divided by everyone who opened the app.

The numbers the ML team watched went up, and the one number the store lives on went down. You can check the last row yourself: orders per visitor, times average order value, times the share not returned. For the old model, 0.020 times 780 times 0.86 is Rs 13.42.

The business head wants to know why a better model earns less. The rest of this guide answers that, and shows how to stop it happening again.

Picture a badminton coach who measures only one thing: smash speed. Every month the player smashes faster, and every month the player loses more matches. The coach is improving a real skill, just not the one that wins.

Smash speed is AUC on taps: real, easy to measure and easy to improve. The match result is net revenue per visitor. Where the picture breaks: a coach sees the result the same evening, while the store sees net revenue only after returns arrive weeks later.

An ML metric measures how well a model predicts on data, while a business metric measures whether the company gets what it wants, such as revenue or loyal customers. The first grades the model, and the second grades the outcome.

ML metric. A number that compares a model's predictions with known answers. AUC, precision (the share of predicted taps that were real taps) and log loss (a penalty that grows when the model is confidently wrong) are common ones. You usually compute them offline, on saved past data, with no live shoppers involved.

Business metric. A number the company is run by, such as net revenue per visitor, profit margin, or how many shoppers buy again within 90 days. You measure it online, with real shoppers using the live app.

Proxy metric. A metric you can measure quickly that you hope moves together with the one you care about. Tap rate is a proxy for buying. A proxy is useful only while that link holds, and the shelf test shows it can break.

You need both. ML metrics are fast and cheap, so the team can compare dozens of model ideas in an afternoon. Business metrics are slow and noisy, but they decide whether the work was worth doing.

AUC measures how often a model scores a random positive example above a random negative one. For the shelf model, it is the chance that a tapped product scores higher than an untapped product.

AUC stands for area under the ROC curve, a plot of the share of tapped items caught against the share of untapped items wrongly flagged, at every possible cut-off. Tom Fawcett's 2006 introduction to ROC analysis states that this area equals the pair-ranking chance. An AUC of 0.5 is a coin toss, and 1.0 means every tapped product beats every untapped one.

The small function below computes AUC by brute force, one pair at a time. It takes a list of scores and a list of labels, where 1 means tapped and 0 means not tapped. If you do not code, skip to the paragraph after it.

from sklearn.metrics import roc_auc_score

def pairwise_auc(scores, labels):
    pos = [s for s, y in zip(scores, labels) if y == 1]
    neg = [s for s, y in zip(scores, labels) if y == 0]
    wins = 0.0
    for p in pos:
        for n in neg:
            if p > n:
                wins += 1      # tapped item ranked higher
            elif p == n:
                wins += 0.5    # a tie counts as half
    return wins / (len(pos) * len(neg))

scores = [0.9, 0.7, 0.4, 0.3]
labels = [1, 0, 1, 0]
print(pairwise_auc(scores, labels))                    # 0.75
print(pairwise_auc([s / 2 for s in scores], labels))   # still 0.75
print(roc_auc_score(labels, scores))                   # 0.75 from scikit-learn

What just happened? Of the four tapped-versus-untapped pairs, three put the tapped item higher, so AUC is 3 out of 4, or 0.75. scikit-learn's roc_auc_score gives the same 0.75; on 200 random lists with ties, our function and scikit-learn 1.9.1 always matched.

Look at what the function never reads. It has no price, no margin, no return rate and no shelf size. It only asks whether each tapped item sits above each untapped item.

Pause and predict: what happens to AUC if every score is halved? It stays 0.75, because the order does not change. In our tests, any change that keeps the order, such as halving or cubing every score, left AUC exactly the same.

So AUC cannot tell you whether a score of 0.3 really means a 30% chance. A model whose scores match real chances is called calibrated. Fawcett notes that a classifier need not produce calibrated probability estimates for ROC analysis to work.

• Value: a tap on a Rs 199 T-shirt and a tap on a Rs 1,499 kurta count the same.
• Position: a wrongly ordered pair at rank 300 costs as much AUC as one in the top eight that shoppers see.
• Calibration: AUC stays the same whether scores are true chances or just a sorted order.
• What happens after the tap: a purchase, a return and an abandoned cart all look the same, because the label stops at the tap.

A higher AUC leads to lower revenue when the model ranks the wrong outcome well, or when its gains land where they earn nothing. Six ways this happens are below, and the shelf model hit several of them at once.

The label was a proxy

The label was a tap, not a purchase the shopper kept. The model learned what grabs attention: bright colours, big discounts, Rs 199 price tags. In this store, those items are tapped often and returned often.

A Booking.com team reported the same trap in a 2019 paper on lessons from about 150 production models. One reason a better model added no value was over-optimising a proxy metric, such as clicks, that fails to turn into conversions, visits that end in a booking.

AUC is blind to value

Getting the order of two Rs 199 T-shirts right earns as much AUC as getting two Rs 1,499 kurtas right. The business cares far more about the kurtas. So a model can gain AUC on cheap items, lose it on costly ones, and still raise the total.

We tested this on a made-up catalogue of 2,000 items where cheap items get more taps but fewer kept sales. A model that knew each item's true tap chance scored AUC 0.711, not 1.0, because taps themselves are partly random. Its top eight held Rs 2.89 of expected kept margin per shelf view.

Ranking by true expected margin instead scored AUC 0.397 on taps, worse than a coin toss. Its top eight held Rs 28.61 per view, about ten times more. The best model for taps was a poor model for money: a random shelf of eight averaged Rs 6.40.

AUC scores the whole list, but shoppers see eight

AUC counts every pair across the whole catalogue, while a phone screen shows eight products. In the same made-up catalogue, we sorted ranks 201 to 2,000 perfectly and swapped the top eight for weaker items from ranks 9 to 60.

AUC rose from 0.677 to 0.708. Expected taps on the shelf fell by about a quarter, from 0.73 to 0.56 per view. Metrics that score only the first k slots, called top-k metrics, catch this; plain AUC does not.

The test data was shaped by the old model

Last month's taps only happened on products the old model chose to show. Products it hid had no chance to be tapped. So offline data quietly favours models that agree with the old one.

Gomez-Uribe and Hunt's 2015 paper on the Netflix recommender says offline experiments assume members would have behaved the same way under the new algorithm. The authors add that they do not find offline results as predictive of A/B test outcomes as they would like.

The win was taken from somewhere else

Some shelf taps come from shoppers who would have found the same item through search anyway. The shelf looks better and the app as a whole does not. The Netflix paper warns that some local engagement wins simply cannibalise viewing from other parts of the product.

There was little left to gain

Sometimes the old model is already good enough that a better one changes almost nothing for shoppers. The Booking.com team called this saturation of business value. Their broader finding was that increasing a model's performance does not necessarily translate into a gain in value.

Pick one main metric that tracks long-term value and that a test can move within weeks, then add guardrail metrics that must not get worse. The main metric picks the winner, and the guardrails can veto it.

Experiment teams call the main metric the overall evaluation criterion, or OEC. A 2012 paper by Ron Kohavi and colleagues at Microsoft, shown at the KDD data mining conference, tells a sharp story about choosing it. In one Bing experiment, a bug showed users very poor search results.

During that buggy test, distinct queries per user rose by over 10% and revenue per user rose by over 30%. The authors explain that poor results force people to issue more queries and click more on ads. The authors compare it to raising prices in a shop: short-term revenue rises, then customers prefer the competition.

The paper's advice is that lifetime customer value should guide the OEC. The Netflix paper names improved member retention as the main target for its recommendation changes. Because retention is already high and hard to move, Netflix also watches engagement, the time members spend watching, which it found strongly correlated with improving retention.

For the shelf, the team builds a ladder of metrics from the model up to long-term health. Each rung should move the next one, and each can be checked.

Next come the guardrails. A guardrail metric is one you do not try to raise, but you refuse to let it get worse. These are the four the team chose for the shelf (made up):

• Return rate must not rise by more than one percentage point.
• Complaints about size and quality must not rise.
• Home screen load time must stay under two seconds.
• The share of orders from shoppers who bought before must not fall.

Rank products by expected value instead of raw probability: multiply the chance of a purchase by what that purchase is worth after returns. The models still predict behaviour, and your code adds the money.

The team changes two things. It trains a buy model on purchases instead of taps, and a return model that predicts whether a bought item comes back. Then it combines them with each product's margin, the money left after product cost and delivery.

In the code below, p_buy and p_return hold each item's predicted chances from those two models. Each item has a margin and a return_cost in rupees from the store's own price data. It runs on Python 3.7 or later with no extra packages.

from dataclasses import dataclass

@dataclass
class Item:
    id: str
    margin: float        # rupees kept on a sale
    return_cost: float   # rupees lost when it comes back

def expected_margin(item, p_buy, p_return):
    kept = p_buy * (1 - p_return) * item.margin
    lost = p_buy * p_return * item.return_cost
    return kept - lost

def fill_shelf(items, p_buy, p_return, slots=8):
    scored = []
    for item in items:
        value = expected_margin(item, p_buy[item.id], p_return[item.id])
        scored.append((round(value, 2), item.id))
    scored.sort(reverse=True)
    return scored[:slots]

items = [Item("tshirt", 40, 60), Item("kurta", 300, 60)]
p_buy = {"tshirt": 0.06, "kurta": 0.03}      # from the buy model
p_return = {"tshirt": 0.30, "kurta": 0.10}   # from the return model
print(fill_shelf(items, p_buy, p_return))   # [(7.92, 'kurta'), (0.6, 'tshirt')]

The kept line is the money the store makes when a shopper buys and keeps the item. The lost line is the pickup and repacking cost when it comes back. The output puts the kurta first, even though the T-shirt is twice as likely to sell.

What just happened? The T-shirt has a 6% buy chance, Rs 40 margin, 30% returns and Rs 60 return cost, so it is worth about Rs 0.60 per view. The kurta has a 3% buy chance, Rs 300 margin, 10% returns and Rs 60 return cost: about Rs 7.92.

Calibration starts to matter

Once you multiply scores by rupees, the size of each score matters, not only its order. Scaling every buy chance by the same factor, such as halving, never changes which items win. Squashing or stretching the scores can.

Take two items. Item A has a 10% real buy chance and is worth Rs 20 per sale. Item B has a 2% chance and is worth Rs 80. A is worth Rs 2.00 per view and B Rs 1.60, so A should win.

Now give the model scores that are the square root of the true chance: 0.316 and 0.141. The order is the same, so AUC is the same, but B now wins, Rs 11.31 to Rs 6.32.

In a made-up catalogue of 2,000 items, squashed and stretched scores both kept AUC at 0.717. They cut the true value of the top eight by 5% and 25%. AUC could not see either loss.

To check calibration, group predictions into bands and compare the average score in each band with the share that really happened. scikit-learn's calibration_curve does this. The code below builds a model whose scores are the square root of the true chance, the squashing that flipped A and B above.

import numpy as np
from sklearn.calibration import calibration_curve
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
true_chance = rng.uniform(0, 0.2, 100_000)   # made-up buy chances
bought = rng.random(100_000) < true_chance
score = np.sqrt(true_chance)                  # squashed: same order, scores too high

print(round(roc_auc_score(bought, true_chance), 3))   # 0.685, AUC of the truth
print(round(roc_auc_score(bought, score), 3))         # 0.685, the same AUC
actual, predicted = calibration_curve(bought, score, n_bins=4, strategy="quantile")
print(np.round(predicted, 3))   # [0.149 0.273 0.353 0.418] average score per band
print(np.round(actual, 3))      # [0.025 0.074 0.126 0.174] share that bought

Expect 0.685 twice, then two rows of four numbers. In every band the average score is 2.4 to 6 times the real buy rate. AUC saw nothing wrong; the calibration check did. We ran it on scikit-learn 1.9.1 and NumPy 2.5.4, the latest releases as of 19 October 2026.

The team also adds a value-weighted offline metric next to AUC: the margin from kept purchases that landed in the top eight slots, replayed on saved data. It is still offline and still biased by the old model, so it filters ideas rather than deciding launches.

You test it with a controlled A/B test: show the old and new model to random groups of shoppers, then compare the main business metric and every guardrail. Offline metrics only decide which models are worth testing.

This is how the Booking.com team worked. Its paper describes about 150 customer-facing models validated through randomised controlled trials, the formal name for A/B tests.

How many shoppers does the test need?

Revenue per visitor is a noisy number, because about 98 in 100 visitors buy nothing. Pause and predict: how many shoppers per group would you need to see a 2% change?

We simulated the store's old numbers: 2% of visitors order, orders average Rs 780 with a standard deviation of Rs 500, and 14% are returned. The standard deviation is the typical distance from the average. Net revenue per visitor averaged Rs 13.44, with a standard deviation of about Rs 121.

A standard two-group formula gives shoppers per group as 15.7 times the variance, divided by the difference squared. The variance is the standard deviation squared. The 15.7 comes from 95% confidence and 80% power, the chance that the test spots a real change of that size.

A simulation at the 6% size detected the change in about 78% of 8,000 runs, a little under the 80% target. Our 6% lift makes every order 6% bigger, which also widens the new group's spread, and the simple formula assumes equal spreads. Treat the formula as a starting estimate, not a promise.

Here is how the team runs the second test, with the expected-margin model. Every step guards against one of the traps above.

1. 1

   Write the decision rule first

   Before the test starts, the team and the business head agree: ship only if net revenue per visitor rises and no guardrail breaks.

2. 2

   Size the test

   Use the formula above to pick the smallest change worth finding. Here the team aims for 6%, which needs about 3.6 lakh shoppers per group, so the test runs four weeks (made up).

3. 3

   Split shoppers, not visits

   Each shopper stays in one group for the whole test. Otherwise one person sees both shelves and the results blur together.

4. 4

   Wait for returns

   In this store, shoppers can return items up to 15 days after delivery (a made-up policy). Judge net revenue only after that window closes for most orders.

5. 5

   Read every rung of the ladder

   Check taps, orders kept, net revenue and guardrails together. A win on one rung with a loss on the next means something broke between them.

6. 6

   Rerun surprising wins

   If a metric jumps with no clear reason, rerun the test on a bigger sample with freshly randomised shoppers. In one Bing case, surprising effects vanished on a larger rerun; the paper traces such puzzles to carryover, users still reacting to an earlier experiment. The authors also recommend A/A tests, both groups on the old model, to check the split.

The second test goes differently. Shelf taps rise only 3%, and AUC on taps is lower than the tap model's. But net revenue per visitor rises about 6% and returns fall.

A loss on the first try is normal, not unlucky. Kohavi and colleagues, citing their 2009 paper on online experimentation at Microsoft, report that only one third of ideas tested there improved the metrics they were designed to improve. Expect many model ideas to fail the business test.

Explain it as a chain: the model metric should predict a proxy, the proxy should predict the business metric, and an A/B test checks each link. Then show where the chain can break.

1. 1

   Start from the business goal

   Say what the company wants, such as net revenue per visitor or repeat buyers. Name two or three guardrails.

2. 2

   Pick the label carefully

   Explain why you predict purchases kept, not clicks. Mention that clicks are easy to collect but easy to game.

3. 3

   Choose offline metrics

   Use AUC or log loss for the model, plus a top-k metric, which scores only the first k slots, such as value in the top eight.

4. 4

   Connect scores to money

   Rank by expected value: chance of purchase times margin, minus expected return cost. Add that this needs calibrated scores, not just a good order.

5. 5

   Plan the A/B test

   Describe random user splits, the main metric, guardrails, the sample size, the test length and the rule for shipping.

6. 6

   Name the traps

   Mention offline data shaped by the old model, wins taken from other pages, and short-term gains that hurt long-term trust.