A/B Testing LLM Applications: Metrics, Splits and Safe Rollout

You choose the text that reads best to you, but real users may behave differently. This is the classic trap when testing LLM applications: a prompt that sounds better in a meeting can sell less, cost more or invent product details.

Picture a common setup. An online fashion store in Surat sells sarees, kurtis and dress material, and a model writes the description for each of its 6,000 products. The store and every number about it are made up, but the problem is real.

The live prompt, prompt A, asks for a 60-word description covering fabric, colour, sizes and care. The marketing lead wrote prompt B: 80 words that open with an occasion the outfit suits, then give the same facts.

He read 20 descriptions from each prompt, loved B and wanted to switch everything that day. Liking 20 texts in a meeting is not the same as more shoppers adding kurtis to their cart. This guide follows how the team finds out.

You need six terms. Each one gets a plain meaning and an example from the store.

Variant. A variant is one version of the thing you test. Version A is usually what runs today, often called the control, and version B is the new idea, often called the treatment.

Offline evaluation. Offline evaluation means scoring a variant on a fixed set of saved examples, with no real user involved. Example: the team runs both prompts on 200 old products and checks each text against the product's real details.

Online A/B test. An online A/B test shows each variant to a different random group of live users and measures what they do. Example: for 14 days, half the shoppers see B's descriptions and half see A's.

Primary metric. The primary metric is the one number you agree, in advance, will decide the winner. Example: add-to-cart rate, the share of shoppers who open a product page and then add an item. If 400 of 10,000 shoppers add an item, the rate is 4%.

Guardrail metric. A guardrail metric is a number that must not get worse, even if the primary metric improves. Example: the share of orders returned as "not as described". A prompt that sells more by overselling the fabric is not a win.

Randomisation unit. The randomisation unit is the thing you assign to group A or B: a user, a session, a page view or a product. Example: the team assigns each shopper, so one shopper sees the same prompt's text on every page.

A/B testing LLM applications works by giving version A to one random group of users and version B to another, then comparing what the groups do. The versions can differ in the prompt, the model or the settings.

Before any real user sees B, it climbs a ladder of cheaper checks. Offline evaluation asks whether B is correct and safe enough to show anyone. The online test asks whether B changes what users do. You need both, in that order.

Think of a new dish at a dhaba. The cook tastes it in the kitchen first: is the salt right, is anything burnt? That is offline evaluation. Then some tables get the dish, and the owner counts who orders it again. That is the online test.

Where the picture breaks: a dhaba owner sees each table's face, but an A/B test only sees clicks and orders. It cannot tell you why shoppers liked B, only that their behaviour changed.

The team's offline check uses 200 past products with known details. Does each description name the right fabric? Does it invent claims like "pure silk" for a silk blend? Is it the right length? The SevyDevy guide on LLM evaluation metrics covers how to build sets like this.

Prompt B passes with one fix. In 6 of the 200 texts, it called a polyester saree "perfect for summer", a comfort claim the team did not want. One added line in prompt B bans comfort claims for synthetic fabric.

Choose one primary metric that the change can move, that you can measure within the test, and that the business cares about. Then add a guardrail metric for each harm this change could cause.

The team tried three candidates. Revenue sounded best, but a few big wedding lehenga orders can move it by several lakh rupees (a lakh is 100,000) in one day. That noise would hide a small real effect.

Time on page was easy to measure, but an 80-word text keeps people reading longer even if nobody buys. So they chose add-to-cart rate per shopper. A description can change it, it moves within days, and it leads to sales.

The store's add-to-cart rate today is about 4.0%. Limits below are in percentage points, the plain difference between two percentages: a rise from 1.0% to 1.3% is 0.3 percentage points.

Next come the guardrails. Ask: if prompt B "wins" in a bad way, what would that look like? Here is the team's list, each with a made-up limit written down before the test:

• "Not as described" returns: must not rise by more than 0.3 percentage points. Text that oversells the fabric will show up here. Returns arrive late, so this one is judged after the return window closes.
• Complaints about wrong product details: must not rise by more than 0.1 percentage points.
• Cost per description: B writes longer text, so it produces more tokens, the small pieces of text a model reads and writes, and costs more. It may cost at most Rs 0.20 more than A.
• Page load time: must not get more than 50 milliseconds slower. This matters most if text is written live, while the shopper waits.
• Bounce rate, the share of shoppers who leave after one page: must not rise by more than 0.5 percentage points.

Split by user, using a stable hash of the user's ID. A hash is a function that turns any text into a number that looks random but is always the same for the same text. So each person lands in the same group every time.

The team looked at three randomisation units. By page view, a shopper could refresh, see a different text for the same kurti and land in both groups. By product, sarees might all land in B and kurtis in A, so the groups differ before the test starts.

By shopper, each person gets one consistent experience. There is one more LLM-specific choice. A model can write a different description each time you ask, even with the same prompt.

So the team does not write text live on each page view. It writes each product's description once with prompt A and once with prompt B, stores both, and shows the stored text that matches the shopper's group.

Here is the assignment code. The helpers log_exposure and load_description are made up for this guide. The first saves a record that a shopper saw a variant. The second fetches a stored description from the database.

import hashlib

EXPERIMENT = "desc-prompt-b"

def pick_variant(shopper_id, percent_b=50):
    key = f"{EXPERIMENT}:{shopper_id}".encode()
    bucket = int(hashlib.sha256(key).hexdigest(), 16) % 100   # 0 to 99
    return "B" if bucket < percent_b else "A"

def description_for(product_id, shopper_id):
    variant = pick_variant(shopper_id)
    log_exposure(shopper_id, product_id, variant)     # who saw what
    return load_description(product_id, variant)      # stored, not live

The key joins the experiment name and the shopper's ID, such as "desc-prompt-b:S1042". Adding the experiment name means a shopper's group here does not decide their group in the next experiment.

The sha256 line hashes the key with a hash function that ships with Python, and keeps a number from 0 to 99, called the bucket. Buckets below percent_b get B, the rest get A. No table of assignments is needed.

You can check this yourself. Over 100,000 made-up IDs, S0 to S99999, the code put 49.9% in group B. Calling pick_variant("S1042") returned "A" every time, and with a second experiment name, half of group B moved to A.

The "bucket below percent_b" rule also makes rollout safe. Raising percent_b only adds shoppers to B: in the same 100,000 IDs, every shopper in B at 50% was still in B at 95%. Lowering it does the opposite. Going from 50 to 10 sent 80% of group B back to A.

Finally, log_exposure records who saw which variant. You analyse that log later, so only shoppers who actually saw a description count. The last line loads stored text, so the model is never called while a shopper waits.

You need enough users to tell a real difference from random noise, and the smaller the difference, the more users you need. Then run for whole weeks, never less than one full week.

Here is the intuition. Toss a coin 10 times and 7 heads tells you almost nothing. Toss it 10,000 times and 7,000 heads would be a real surprise. With a few hundred shoppers per group, rates jump around by chance in the same way.

The marketing lead says a rise from 4.0% to 4.4% add-to-cart rate is worth having. That gain of B over A is the lift, here 0.4 percentage points. The standard sample size formula says this needs about 39,500 shoppers in each group.

That uses the usual settings. If B is truly no better, the test has a 5% chance of a false alarm, counting both directions together. If the lift is truly 0.4 points, it has an 80% chance of detecting it.

You can check the number with statsmodels, a free statistics library for Python. We tested it with statsmodels 0.15.0 on Python 3.13 in October 2026. Run this and expect 39454 shoppers per group:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# h turns the two rates into one standard effect size
h = proportion_effectsize(0.044, 0.040)   # 4.4% vs 4.0%
print(round(NormalIndPower().solve_power(h, alpha=0.05, power=0.8)))
# prints 39454 (shoppers per group)

A handy rule follows from the formula: to catch a lift half as big, you need about four times as many users. For a lift to 4.2%, the same formula gives about 154,300 shoppers per group, 3.9 times more.

About 8,000 shoppers a day visit the store's product pages, a made-up number. About 79,000 shoppers in total means about 10 days. Some shoppers come back on later days and count only once, so the team rounds up to 14 days.

Fourteen days is exactly two weeks, so every day of the week appears exactly twice. The 4.2% target would need about 39 days at 8,000 shoppers a day, before allowing for returning shoppers. Small effects cost a lot of time, so decide what size of win matters before you start.

We simulated 20,000 such tests, each with 4,000 new shoppers per group per day at a 4% rate. Looking once, on day 14, gave a false win 4.8% of the time.

Checking every day and stopping at the first significant result gave a false win 21.7% of the time, more than four times as often. About half of those false wins favoured B and half favoured A. Significant here means a two-proportion z-test, the usual check for whether two rates differ by more than chance, gave a p-value below 0.05.

A p-value is the chance of seeing a gap at least this big if A and B were truly the same. Each daily look is another chance for noise to cross that line.

So pick the end date before you start. If you must check early, use sequential testing, a family of methods built for it. One example, always-valid p-values, stays correct however often you look.

It is harder because the model adds its own randomness, can change under you, and costs money per answer. The usual problems of A/B tests, like novelty effects, still apply on top.

Non-determinism. A program is non-deterministic when the same input can give a different output. Models usually pick each next token partly at random, called sampling. One 2024 study found outputs still varied for the same input even under settings configured to be deterministic.

Example: ask prompt B twice about the same green kurti and you may get "perfect for Navratri" once and "ideal for office days" the next.

Novelty effect. A novelty effect is when users react to something because it is new, and the reaction fades as they get used to it. The opposite, where engagement grows as users adopt a change, is called a primacy effect.

Here is how each problem shows up in an LLM test, and what the team does about it:

• Random text: written live, one kurti could show a shopper two different texts. Storing each text once turns the test into two fixed sets of text.
• Lucky texts: one odd description on a best-selling saree could sway the result. Thousands of products spread that luck out.
• A model that changes: if the provider updates the model mid-test, the result mixes two models. Pin a dated model version where one is offered, and log the version either way.
• Novelty: compare B's lift in week 1 with week 2, and new shoppers with returning ones. A lift that only appears among returning shoppers in week 1 is a warning sign.
• Cost and speed: B's longer text costs more and takes longer to write. Because texts are stored, shoppers never wait, but cost stays a guardrail.

Shadow testing runs the new variant on real live inputs but never shows its output to users, so you can check it safely. Interleaving mixes two ranked lists into one list for the same user, which can need fewer users than an A/B test.

How shadow testing works

Before the A/B test, the team ran prompt B in shadow for one week. Every new product, about 300 that week, got a description from both prompts. Shoppers only ever saw A's text.

B's text went into a log, where automatic checks and a copywriter looked at it. The diagram shows the two paths.

The shadow week caught one real problem. For products with a missing colour field, prompt B guessed a colour instead of leaving it out. The 200 offline test products all had colours, so the offline check had missed it.

The team now sends such products to a copywriter instead. Shadow testing also showed B's real cost: about Rs 0.30 per description, Rs 0.12 more than A and inside the Rs 0.20 guardrail. It cannot show whether shoppers prefer B, because no shopper saw it.

When interleaving helps

Interleaving works when the output is a ranked list, like search results. A ranker is the code that decides the order of that list. Later, the store tested a new search ranker for queries like "cotton kurti".

Instead of giving each group its own list, every shopper saw one mixed list. Items came from ranker A and ranker B in turns, and a coin toss decided which went first. Each click was credited to the ranker that supplied that item.

Each shopper compares both rankers at once, so differences between people matter less. One theoretical study found interleaving more efficient than A/B testing when users leave a list depending on how relevant its items are.

Interleaving does not fit product descriptions, because a page shows only one description. It also tells you which ranker gets more clicks, not how much sales change, so a final A/B test still decides.

Check the split first, then the primary metric's range, then every guardrail, using rules you wrote down before the test. Roll a winner out in small steps, with a switch that brings back the old version.

After 14 days, using made-up numbers, group A had 41,200 shoppers and group B had 40,900. Add-to-cart rate was 4.02% for A and 4.38% for B.

The 95% confidence interval for B's lift was +0.09 to +0.63 percentage points. A confidence interval is the range of true lifts that are consistent with the data you saw. The whole range is above zero, so chance alone is a poor explanation (two-sided p = 0.01).

The team wrote its checking order as a function, so the decision does not depend on who is most excited in the meeting. Two helpers are made up. split_looks_wrong compares group sizes with the planned 50/50. compare returns the range for B minus A on one metric.

GUARD_LIMITS = {"not_as_described_rate": 0.003, "complaint_rate": 0.001,
                "cost_per_text": 0.20, "page_load_ms": 50,
                "bounce_rate": 0.005}

def decide(test):
    if split_looks_wrong(test, expected_b=0.5):   # groups not about 50/50
        return "stop: find the bug, then rerun"
    cart = compare(test, "add_to_cart_rate")      # range for B minus A
    if cart.low <= 0:
        return "no clear win: keep prompt A"
    for name, limit in GUARD_LIMITS.items():
        guard = compare(test, name)
        if guard.high > limit:                    # B might be too much worse
            return f"blocked by guardrail: {name}"
    return "ship B with a slow rollout"

GUARD_LIMITS holds how much worse each guardrail may get. Rates are fractions, so 0.003 means 0.3 percentage points. Cost is in rupees and page load in milliseconds.

No limit is exactly zero. Noise always gives a range some width, so a zero limit would block B unless it were clearly better on that guardrail too.

The first check is the split. Group sizes far from the planned 50/50 are called a sample ratio mismatch. It is a symptom of a data problem, such as a page that forgot to log one group, so find the cause first.

A chi-square test checks whether counts match the split you planned. You can run one with SciPy 1.18.1, a free science library for Python. A tiny p-value means the gap is too big to be chance:

from scipy.stats import chisquare

print(round(chisquare([41200, 40900]).pvalue, 2))   # prints 0.3: a normal gap
print(chisquare([41200, 39900]).pvalue)           # prints about 5e-06: broken split

Next, cart.low is the bottom of the primary metric's range. If it is zero or below, B might be no better than A, so A stays. Then the loop blocks B if any guardrail's range could break its limit. Returns lag delivery, so the team runs decide two weeks after the test ends (made up), once the return window closes.

We ran decide with stand-in helpers, these numbers and made-up guardrail ranges inside their limits. It returned "ship B with a slow rollout". The lift was smaller in week 2 than week 1, a mild novelty effect, but still above zero. Before trusting any result, the team runs this honesty list:

• The primary metric, guardrails and end date were written down before the test, and never changed.
• The test ran to its planned end, with no early stop.
• The split was checked before anything else.
• Nobody searched through 30 other metrics or shopper groups for one that happened to look good.
• The report gives the range, not just one number, and a flat result is reported as flat.

Then the rollout. The team does not switch everyone to B at once. It raises percent_b in the same assignment code and never lowers it:

1. 1

   Keep 50% until the decision

   The test's split stays in place while late returns arrive, so no shopper changes group. Restarting at 10% would send most of the test's B shoppers back to A.

2. 2

   Go to 95%

   Move up only if every guardrail stayed inside its limit. It is one settings change, and shoppers already on B stay on B.

3. 3

   Keep the switch back

   New products still get a checked text from both prompts for a month. One setting can then send every shopper back to A.

4. 4

   Keep a small holdback

   The last 5% of shoppers keep seeing A for a few more weeks. This holdback group shows whether the lift lasts once the novelty wears off.

Explain it as a ladder of tests, each showing the change to more real users, with decision rules fixed before the online test starts. Interviewers want to hear how you avoid fooling yourself, not just how traffic gets split.

1. 1

   State the decision and the metric

   Say what you are choosing between, like two prompts, and name one primary metric and two or three guardrails.

2. 2

   Test offline, then in shadow

   Score B on saved examples, then run it on live inputs without showing it, to catch errors and measure cost.

3. 3

   Split by user with a stable hash

   Explain the randomisation unit, and how you store or log outputs so model randomness does not leak into the test.

4. 4

   Size and time the test

   Give rough user numbers for the smallest lift worth having, and run whole weeks without stopping early.

5. 5

   Decide, then roll out slowly

   Check the split, the primary metric's range and every guardrail, then ramp up with a switch back to A.