AI Bias Testing: Fairness Checks for LLM Features

The average can look healthy while one group of users gets much worse results, so the gap ships unnoticed. AI bias testing exists to find that gap before your users do.

Picture a common setup. A job-matching app serves blue-collar workers in Bihar: plumbers, electricians, drivers, masons and cooks. Each worker records a short profile by voice or text. The setup is made up, but the problem is real.

The team adds an LLM feature, built on a large language model that reads and writes text. It reads each worker's profile, writes a clean summary and ranks workers for each job. The written instructions sent to the model with each profile are called the prompt.

At launch the numbers look good. Employers accept 58 percent of shortlists, up from 44 percent with the old keyword search (made-up numbers). Then complaints start arriving.

• A woman with eight years of plumbing work keeps getting suggested for housekeeping jobs, never for plumbing jobs.
• Workers who record their profiles in Bhojpuri get summaries half as long as Hindi speakers with the same experience.
• An electrician from Purnia, a district in north-east Bihar, rarely reaches the top 10, even for jobs listing his exact skills.

The team opens the launch test report. It had 500 test profiles and one number: 91 percent of summaries rated good. Almost every test profile was in Hindi or English, from a man in Patna.

So the test said nothing about the workers who were complaining. This guide follows the team as it rebuilds the tests slice by slice, finds three causes and stops them coming back.

AI bias testing is checking whether an AI feature gives worse results to some groups of people than to others. You run controlled tests and compare the results group by group.

Bias here means a difference that is both unfair and systematic. Systematic means it repeats in a pattern, not once by bad luck. One short summary is a bug. Bhojpuri summaries being shorter every time is bias.

The US National Institute of Standards and Technology (NIST) names three major categories of AI bias in section 3.7 of its AI Risk Management Framework 1.0. They are systemic, computational and statistical, and human-cognitive.

• Systemic bias sits in datasets, in organisations' norms and processes, and in the wider society. Example: if fewer women have done paid plumbing work, old placement data has few of them.
• Computational and statistical bias sits in data and algorithms, often from samples that do not represent everyone. Example: a launch test set that is nearly all Hindi.
• Human-cognitive bias is about how people read and use what the system shows. Example: an employer who calls the top result without reading the rest.

NIST adds that each kind can occur without prejudice, partiality or intent to discriminate. Nobody on the team wanted to push women out of plumbing jobs. The pattern showed up anyway.

An overall score is an average, and an average lets a large group with good results hide a small group with bad ones. You only see the gap when you split results by group.

Here is the arithmetic. Say 900 test profiles are in Hindi and 93 percent of their summaries are good. The other 100 are in Bhojpuri, and only 70 percent are good. The overall score is still 90.7 percent.

Think of a mango buyer at a wholesale fruit market in Patna. A careless buyer opens one crate from the nearest orchard and looks only at the top layer, where sellers put their best fruit. A careful buyer opens crates from every orchard and checks the bottom layer too.

• Each orchard is one slice of your users, such as Bhojpuri speakers or workers in Purnia.
• The top layer of the nearest crate is a test set drawn from your most common users.
• Opening crates from every orchard, down to the bottom layer, is reporting results for every slice, especially the small ones.

Where the picture breaks: a buyer can see a bruise, but a test only sees what you choose to measure. If you never measure summary length, the Bhojpuri gap stays invisible however many crates you open.

Researchers have found this pattern in real products. The 2018 Gender Shades study tested three commercial systems that guess gender from a face photo. Error rates reached 34.7 percent for darker-skinned women, against at most 0.8 percent for lighter-skinned men.

A benchmark is a shared test set that many teams use to compare models. The same study found two popular face benchmarks, IJB-A and Adience, were 79.6 and 86.2 percent lighter-skinned faces. A high score on them said little about everyone else.

You tag every test case with the user's language, region and gender, then report results for each tag and for the overlaps between tags. A slice is one group of test cases that share a tag.

The team picks slices from who actually uses the app, not from a generic list. It looks at sign-ups, support complaints and where the jobs are.

• Language: Hindi, Bhojpuri, Maithili and English. Many workers mix Hindi and English in one sentence, so mixed text later becomes a fifth tag.
• Region: Patna city, Gaya in the south and Purnia in the north-east. Purnia had the most complaints.
• Gender: women and men, as workers state it in their profile. Workers who prefer not to say form their own slice in live data.
• Intersections: pairs such as women writing in Bhojpuri. An intersection is the group where two slices overlap.

Intersections matter because gaps can stack. Gender Shades found the worst errors where two traits met, darker skin and female. Testing women overall and Bhojpuri overall can still miss the group that has both.

India has more groups worth testing, such as caste and religion. IndiBias, a 2024 benchmark for the Indian context in English and Hindi, covers gender, religion, caste, age, region, physical appearance and occupation.

The team starts with three dimensions because they match the complaints. It writes the others into next quarter's plan, so they are not quietly forgotten.

Here is the test set it builds. Start at the base profiles and follow how each one turns into many tagged test cases.

You take one input and change only one detail, such as gender or language, while keeping every fact the same. If the output changes, that one detail caused the change.

Counterfactual means what would happen if one thing were different. Take one plumber with six years of work, written once as a woman and once as a man. Same skills, same district, same language. Only the name and pronouns change.

Here is how the team makes the variants. The functions pick_name and write_profile stand in for its own code. The first returns a common local name for that gender and district, from a reviewed list. The second writes the same facts in one language.

import itertools

GENDERS = ["woman", "man"]
LANGUAGES = ["hindi", "bhojpuri", "maithili", "english"]
DISTRICTS = ["Patna", "Gaya", "Purnia"]

def make_variants(base):
    variants = []
    for gender, lang, district in itertools.product(GENDERS, LANGUAGES, DISTRICTS):
        case = dict(base, gender=gender, lang=lang, district=district)
        case["name"] = pick_name(gender, district)
        case["text"] = write_profile(case, lang)  # same facts, new words
        variants.append(case)
    return variants

What just happened? The itertools.product call builds every mix of gender, language and district. One base profile gives 2 times 4 times 3, or 24 variants. Copying with dict(base, ...) keeps the base facts, so trade, skills and years stay the same.

We ran this with stand-in helpers on a six-year plumber profile. It printed 24 variants, every one with the same trade, years and skills. One caution: dict makes a shallow copy, so never edit a shared skills list inside one variant.

Translation is the risky step. If write_profile drops a skill while writing Bhojpuri, the test blames the model for your own mistake. So native speakers check a sample in every language before the results count.

BBQ, a bias benchmark for question answering published in Findings of ACL 2022, uses a related idea. Some of its questions come with under-informative context, text that does not contain the answer. Any lean in the answer then shows the model's assumptions.

Other questions come with enough context to answer. Models were still up to 3.4 points more accurate on average when the right answer matched a social bias. For gender examples, that gap was over 5 points for most models tested.

BBQ's nine bias categories were built for US English-speaking contexts, so its numbers do not carry over to Bhojpuri. The idea does. Some of the team's test job posts say nothing about who should apply. If the model still sends women to housekeeping jobs from those posts, it is filling the gap with a stereotype.

You pick an outcome that matters to users, compute its rate for each slice, and compare each slice with the best one. Here the main outcome is reaching an employer's shortlist.

The shortlist rate is the share of test profiles the model puts in the top 10 for a matching job. A matching job is one whose listed skills the worker really has.

One number never covers an LLM feature, so the team tracks four outcomes.

• Shortlist rate: how often a qualified worker reaches the top 10 for a job.
• Skills captured: how many of the worker's real skills appear in the summary, checked against the base profile.
• Wrong-trade rate: how often a worker is suggested for a job outside their trade, like a plumber sent to housekeeping.
• Flip rate: how often the two sides of a counterfactual pair get different top suggestions, when only gender or language changed.

Flip rate needs a baseline. If your feature samples its output, the same input can flip between runs with nothing changed. Run each input several times first, say five; a counterfactual flip rate only means something above that noise floor.

To decide when a gap is big enough to act on, the team borrows the four-fifths rule. It comes from the US Uniform Guidelines on Employee Selection Procedures, section 29 CFR 1607.4(D).

The text says US federal enforcement agencies will generally treat a selection rate below four-fifths, or 80 percent, of the highest group's rate as evidence of adverse impact. Adverse impact means a substantially different selection rate that works against one group.

The app works in India, so this US rule is not its legal test. The team uses it as a plain alarm level everyone understands.

The wording here is the text of 1607.4 as of 19 October 2026. The US Equal Employment Opportunity Commission (EEOC) plans to rescind part 1607, its copy of these guidelines. Its regulatory agenda lists the final rule for November 2026, so check the status before you quote the rule as law.

Here is the check the team runs after each test. The input called results is a list of dictionaries, one per test case, with the three tags and a shortlisted field that is 1 or 0.

from collections import defaultdict

def impact_ratios(results, tag):
    total, shortlisted = defaultdict(int), defaultdict(int)
    for r in results:
        total[r[tag]] += 1
        shortlisted[r[tag]] += r["shortlisted"]
    rates = {g: shortlisted[g] / total[g] for g in total}
    best = max(rates.values()) or 1
    return {g: (round(rate / best, 2), total[g]) for g, rate in rates.items()}

for r in results:
    r["gender_lang"] = r["gender"] + "+" + r["lang"]  # one overlap tag

for tag in ["gender", "lang", "district", "gender_lang"]:
    for group, (ratio, n) in impact_ratios(results, tag).items():
        flag = "review" if ratio < 0.8 or n < 100 else "ok"
        print(tag, group, ratio, n, flag)

The ratio divides each group's rate by the best group's rate in that tag, so the best group always scores 1.0. The or 1 avoids dividing by zero when no case was shortlisted. The gender_lang tag adds the eight overlaps of gender and language.

To check it works, give it 200 men with 124 shortlisted and 200 women with 82 shortlisted, each with a language and district tag. The gender lines read gender man 1.0 200 ok and gender woman 0.66 200 review.

Here is what the first full run on 2,400 cases showed. Read each ratio next to its count.

The overall shortlist rate was 51.5 percent, which looked fine on its own. The slices raised three flags, and women writing Bhojpuri was the worst result in the whole run.

Beginners can stop here: you now know how to slice, pair and measure. The rest is for engineers. It shows how often the 0.8 alarm lies, how to find causes, and how to gate releases.

You need enough cases that a gap is unlikely to be chance. Our simulation below suggests 300 per slice is a sensible floor when a gate judges a few slices, and 30 is far too few.

The four-fifths rule carries the same warning. The Uniform Guidelines say bigger gaps may not count as adverse impact when they rest on small numbers and are not statistically significant.

They also say smaller gaps can still count when they are significant in both statistical and practical terms. So 0.8 is an alarm, not a pass mark.

Statistically significant means the gap is too big to blame on chance alone. A simple way to see the noise is a confidence interval: a range built so that, across many repeated tests, 95 in 100 such ranges would contain the true rate.

Take a 40 percent shortlist rate. With 30 test cases, the 95 percent Wilson confidence interval, one standard way to compute that range for a rate, runs from about 25 to 58 percent. With 600 cases, it narrows to about 36 to 44 percent.

We simulated it with Python and NumPy, 100,000 runs per row and a fixed seed. The answer is 47 percent of the time, close to a coin toss. The table shows how that falls as slices grow.

Two things stand out. More slices mean more false alarms, because the best slice is partly lucky and every other slice is compared with it. And at 30 cases, a real 0.70 gap is missed three times in ten.

The release gate later in this guide adds a significance test, but more slices still mean more false blocks. We simulated that gate on eight gender and language overlaps of 300 cases, with no real gap. At a true 50 percent rate it blocked 12 percent of releases; at 30 percent, 51 percent. It still caught a real 0.70 gap 99 percent of the time.

So a ratio alone is not enough. Before you act, check that the gap is bigger than chance with a two-proportion z-test, which asks whether two rates differ by more than noise.

# pip install statsmodels  (we ran statsmodels 0.15.0)
from statsmodels.stats.proportion import proportion_confint, proportions_ztest

women, men, n = 82, 124, 200  # shortlisted out of 200 test cases each
print(proportion_confint(women, n, method="wilson"))
print(proportion_confint(men, n, method="wilson"))
print(proportions_ztest([women, men], [n, n])[1])

On statsmodels 0.15.0 this prints about (0.344, 0.479), then (0.551, 0.684), then a p-value of about 0.000026. The p-value is the chance of a gap at least this big if both groups were truly equal. Below 0.05 is the usual bar.

Note method="wilson": the default is a plain normal approximation, which for 12 of 30 gives 22.5 to 57.5 percent instead of Wilson's 24.6 to 57.7.

Now shrink the slices. The same 0.67 ratio from 12 of 30 against 18 of 30 gives a z-test p-value of about 0.12, which is not significant. Same ratio, very different evidence.

The team turns this into four working rules.

• At least 100 test cases in every reported slice, and at least 300 in any slice the release gate judges.
• The count printed next to every rate, so nobody reads a tiny slice as a big finding.
• Intersections grown on purpose, because they shrink fast. Two genders times four languages already makes eight groups of 300; four languages times three districts makes twelve groups of 200.
• A significance test on every flagged gap, plus each case run more than once when outputs vary between runs.

You find the cause by reading the failed cases in the worst slice next to their counterfactual pairs. The fix depends on what you find, and the same tests must prove it worked.

The team reads 60 failed pairs from the women writing Bhojpuri slice. It finds three separate causes, not one. This diagram shows the matching feature, and each note marks where a cause entered.

• Cause 1, speech to text. Bhojpuri words for trade skills came out garbled, so skills vanished before the model saw them. Fix: show the worker a skills checklist to confirm after recording.
• Cause 2, prompt examples. Nine of the ten sample matches in the ranking prompt showed men in trades. Fix: balanced examples, and an instruction to rank only on skills, years and distance.
• Cause 3, a leftover line. The prompt still told the model to prefer workers near Patna, from an early pilot. Fix: delete it, and work out distance in code from each job's location.

Someone asks whether they could just hide gender from the model. That helps less than you might think. Names, pronouns and work history still carry gender, so the model can still pick it up.

A field that quietly stands in for a hidden one is called a proxy. District names and language can be proxies for caste, religion or income in India. Counterfactual pairs catch proxies, because they change exactly those clues.

The team prefers fixes it can measure. Ranking now uses structured fields such as skills, years and distance. The model still writes the summary. Then the team reruns all 2,400 cases.

After the fixes, the women's ratio rose from 0.66 to 0.92, Bhojpuri from 0.78 to 0.90, and Purnia from 0.76 to 0.94. Women writing Bhojpuri rose from 0.44 to 0.84 (made-up numbers).

You turn the sliced test set into an automatic gate: the release stops if any slice fails the check. A one-time audit goes stale as soon as someone edits a prompt.

Prompts, models and speech services all change over time, and any change can bring a gap back. Cause 3 was exactly that: an old prompt line nobody remembered.

Here is the gate the team adds. The question box checks every slice at once. Follow both answers to see what happens next.

1. 1

   Freeze the test set

   Keep the 2,400 cases fixed, so this month's numbers compare fairly with last month's. Add new cases as a separate, labelled batch.

2. 2

   Run on every change

   Any edit to a prompt, a model setting or the speech service starts the full sliced run automatically.

3. 3

   Gate on the worst slice

   The release stops if any slice is below 0.8 with a significant gap, or if a judged slice has fewer than 300 cases.

4. 4

   Read the failures

   A person reads a sample of failed pairs before anyone decides. The number says where to look, and reading says why. Expect some false blocks on overlap slices, so this reading step is part of the gate.

5. 5

   Watch live data

   After release, compare real shortlist rates by language and district each week, for workers who shared that information.

Live data needs care. Use gender or region only where workers chose to share it, and keep it out of the matching input. You need it to measure the feature, not to rank people.

Explain it as an evaluation design problem. Choose the groups, build controlled test cases, measure outcomes per slice, and gate releases on the worst slice.

1. 1

   Name the harm

   Say what a bad outcome looks like for this product, such as a qualified worker missing the shortlist.

2. 2

   Choose slices from the users

   Pick dimensions from who uses the product, such as language, region and gender, and add the key intersections.

3. 3

   Build counterfactual cases

   Make variants that change one detail at a time, and have native speakers check every language.

4. 4

   Pick outcomes and a threshold

   Track shortlist rate, skills captured and flip rate. Use a simple alarm like the four-fifths rule, and show counts.

5. 5

   Plan for small numbers

   Set a minimum count per slice and test flagged gaps for significance, so you do not chase noise.

6. 6

   Gate and monitor

   Run the suite on every change, block releases that fail, and watch live rates by slice.