Synthetic Data for LLM Evaluation: Test Without Real Customers

Real customer chats are full of personal data, so copying them into a test system spreads that data to laptops, logs and tools never meant to hold it. They also miss the rare cases that break a bot.

Picture a common setup. A health insurer runs a claims chatbot that answers policyholders about claim status, cashless approval (the insurer pays the hospital directly) and rejected claims. The setup is made up, but the problem is real.

Before every release, the QA team (quality assurance, the people who test the software) needs one answer. Did the new prompt (the instructions sent to the model) or a new model make any reply worse? The bot handles about 40,000 chats a month, so the team plans to export 2,000 as its test set.

The privacy officer, who signs off on how personal data is used, opens three random chats. Then she says no.

• A son types his father's full name, policy number and a cancer diagnosis, then asks why chemotherapy was not covered.
• A woman pastes the text of her hospital bill, with her mobile number and address in it.
• A man asks about a claim for his wife's pregnancy, and gives her date of birth to prove who she is.

Health details, policy numbers and phone numbers sit side by side in those chats. Test servers, test reports and shared spreadsheets are exactly where such data tends to leak.

There is a second problem. Most real chats ask for claim status. The cases that break the bot, like one message in Hindi and English about two claims, are rare, so a sample of real chats barely tests them.

This guide shows how the team builds a test set from scratch instead. No real person's data goes in, and the hard cases are made on purpose.

Six words carry the whole guide. Each one gets a plain meaning here, with an example from the claims bot.

• LLM (large language model): the AI that reads and writes text inside the bot. You send it text, called a prompt, and it writes a reply.
• Evaluation: running the app on fixed inputs and scoring what comes out, so two versions can be compared fairly.
• Test case: one input plus a note on what a good reply does. A user asks why a claim was rejected, and a good reply names the reason and the appeal steps.
• Edge case: an unusual input at the limits of what the app handles, such as a policy number with one digit missing.
• Synthetic: produced on purpose from a plan rather than collected from real people. The people, policies and claims in it never existed.
• Regression test: a test you rerun after every change, to check that something which worked before still works.

Synthetic data LLM evaluation means scoring your AI app on a set of made-up test conversations, each stored with the behaviour you expect. A generator model writes the user messages from a plan the team controls.

In the claims bot's set, code also adds fake policy numbers that start with TEST. The insurer has confirmed that no real policy uses that prefix, so a leaked test case is easy to spot.

The idea is older than chatbots. NIST, the United States standards body, names generating synthetic data using models as one de-identification technique. De-identification means removing the links between data and real people.

Synthetic test data is usually safer because it starts with no real person in it, while masked data starts with real people and tries to hide them. Masking has to catch every clue, and it often misses some.

Masking means replacing names, numbers and addresses with placeholders like NAME or PHONE. But a chat can identify someone with no name in it.

Take the son's message from the first section. Remove the name and the policy number. What is left can still say a 71-year-old man in Thane had chemotherapy at a named hospital in March.

Clues like age, area and date are called quasi-identifiers: harmless alone, but pointing to one person when combined. NIST's de-identification guidance lists transforming quasi-identifiers as a technique separate from removing identifiers like names.

Synthetic cases skip this fight. The generator never sees the son's message, only a line from the plan: son of an elderly parent, rejected chemotherapy claim, angry tone. The chat it writes is about a person who does not exist.

There is a practical win too. Singapore's data protection regulator lists software testing as a use case in its proposed guide on synthetic data from July 2024. It says synthetic data can help avoid data breaches if a development system is compromised.

The table hides one honest point. Real chats win on sounding real. The section on limits shows how the team keeps its synthetic chats close to real language without copying anyone.

You generate synthetic test cases in three moves. Write a plan of what to cover, ask a model to write chats from each row of the plan, then filter the results in code. The plan matters more than the generator.

Think of a kabaddi team before a big match. The coach cannot bring the rival side's star raider to practice. So teammates copy his moves: his quick toe touch, his escape to the left, his bonus-line trick.

Map it part by part. The rival raider is your real users, and the coach's notes on his moves are the coverage plan. The teammates copying him are the generator model, and the defenders are your bot.

Where the picture breaks: the coach copies one raider he has watched on video. Your plan copies users nobody on the team has watched closely, so it is only as good as the team's guesses about them.

Step one: build the coverage grid

A coverage grid is a table of the ways your users differ, called dimensions. Every combination of values becomes one row of the plan. The team picks three dimensions for the claims bot.

• Intent, what the user wants: claim status, cashless approval, a rejected claim, adding a family member.
• Persona, who the user is: a first-time user, an angry customer, a son asking for an elderly parent, someone typing in Hinglish (Hindi written in English letters).
• Twist, what makes it hard: none, a wrong policy number, two claims in one message, or asking about someone else's claim.

Four intents, four personas and four twists give 64 rows. The team writes 31 cases per row with different made-up details, which gives 1,984 cases, or about 2,000.

Step two: turn each row into specs

A spec is one row of the plan plus the made-up details for one case. The code below builds all 1,984 specs. It needs only plain Python, and we ran it on Python 3.13.

import itertools
import random

INTENTS = ["claim status", "cashless approval", "rejected claim", "add family member"]
PERSONAS = ["first-time user", "angry customer", "son of elderly parent", "Hinglish typist"]
TWISTS = ["none", "wrong policy number", "two claims at once", "asks about someone else"]

def build_specs(seed=7, runs=31):
    rng = random.Random(seed)                       # same seed, same plan on one Python version
    numbers = rng.sample(range(100000, 1000000), 64 * runs)   # no repeats
    specs = []
    for intent, persona, twist in itertools.product(INTENTS, PERSONAS, TWISTS):
        for _ in range(runs):
            specs.append({"intent": intent, "persona": persona, "twist": twist,
                          "policy": f"TEST-{numbers.pop()}"})   # fake, easy to spot
    return specs

specs = build_specs()
print(len(specs), "specs,", len({s["policy"] for s in specs}), "unique policy numbers")
print(specs[0])

Run it with python3 and it prints two lines. On Python 3.13 the first spec should match exactly:

1984 specs, 1984 unique policy numbers
{'intent': 'claim status', 'persona': 'first-time user', 'twist': 'none', 'policy': 'TEST-982610'}

In the real pipeline, each spec then goes to the generator model along with the insurer's public policy wording. That model call is left out here because it depends on your provider. No real chat is ever pasted in as an example, and the privacy section explains why.

Here is the whole pipeline. Start at the plan and follow the arrows. Notice that people review a sample before the set is used, and that the scorer sits at the very end.

You find edge cases by asking a generator to attack the bot on purpose, keeping the attacks that cause failures, and then making more variants of those. Easy cases teach you little.

Researchers did this at scale in a 2022 paper, Red Teaming Language Models with Language Models. Red teaming means attacking your own system to find its weak spots first. The authors used one model to write test questions for another.

They note that human-written test cases are expensive, which limits how many and how varied they are. A classifier, a model that sorts text into labels, flagged offensive replies. That way they found tens of thousands in a 280-billion-parameter chatbot.

Parameters are the numbers a model learns during training. The authors also compared several ways of writing the questions, from zero-shot generation (asking with no examples) to reinforcement learning (training the generator to find more failures).

For the claims bot, the team asks the generator for edge cases in four families. Each family attacks a different part of the bot.

• Broken input: a policy number with a missing digit, a date written as 31/02, a message cut off halfway.
• Mixed asks: the status of one claim and an appeal on another, in a single message.
• Boundary rules: a claim filed one day after the deadline, or a hospital just outside the cashless network.
• Unsafe asks: requesting a stranger's claim, or asking the bot to approve a claim itself.

The diagram shows what happens to each generated edge case. Follow a failed case: it does not just become a bug ticket. It also becomes the starting point for harder variants.

The realism check is easy to skip, and skipping it hurts. A generator asked for hard cases will happily write nonsense, like a user quoting a section of a law that does not exist. A bot failing on nonsense tells you nothing.

You check synthetic test data in three ways: is each case valid, does the set look like real traffic, and does it actually catch bugs. A set can pass the first two and still miss the third.

Singapore's proposed guide names three similar checks for any synthetic dataset. Integrity means the data is accurate, complete, consistent and valid. Fidelity means it closely follows the source data's characteristics and statistics.

Utility means how well it can replace or add to the source data. For software testing, the guide advises following the source data's semantics, such as formats, minimum and maximum values and categories, rather than its statistics. That fits a test set well, since you want coverage more than lookalike averages. In a set built from a plan, privacy-safe counts of real traffic stand in for the source data.

Validity checks in code

Some checks are exact, so plain code does them almost for free. A case is dropped if it holds a real-looking number, a policy number outside the TEST format, no rule to score, or a near-copy of a kept case. The engineer section shows that code, run on real inputs, and what it still misses.

Realism and usefulness checks

Code cannot tell whether a chat sounds like a real policyholder. So the QA team reads 1 case in 10. They mark each one realistic or not, and check that its rules match the insurer's actual policy terms.

Usefulness is tested on the bot itself. The team keeps an old bot version with three known bugs, all since fixed. If the synthetic set does not catch all three, it is not ready.

Once the set is ready, scoring it is the next job. Choosing metrics, rules-based checks and model judges is its own topic, covered in our guide to LLM evaluation metrics and test frameworks.

Yes, synthetic data can still leak real information if the generator copies a real record or learned from real data. It is safer than real data, not automatically safe.

Singapore's proposed guide says this plainly. Synthetic data may not always be risk-free, because information can still leak when it resembles the source data.

That guide names two attacks to test for. Membership inference means working out whether a person is in the source data. Attribute inference means learning an undisclosed detail about them, such as a diagnosis. It also warns that outliers, people who stand out in the data, are especially easy to single out.

Language models add their own risk. A 2020 paper, Extracting Training Data from Large Language Models, attacked a public language model trained on web text. The authors recovered hundreds of exact passages from its training data.

Those passages included public names, phone numbers and email addresses. Some could be extracted even though each appeared in just one document of the training data. Larger models were more open to this than smaller ones.

This is why the privacy officer bans real chats from the generator. If a model were fine-tuned, meaning trained further on your own data, on real claims chats, it could later repeat one inside a synthetic case.

Pasting real chats into the prompt as examples carries a similar risk, because a model can copy details it was shown. OWASP, a non-profit that publishes security guidance, lists sensitive information disclosure as LLM02 in its 2025 Top 10 for LLM apps.

Its first listed defence is data sanitization, so that user data does not enter model training. The 2025 list was the newest edition on OWASP's site as of 19 October 2026.

The privacy officer adds a gate before any set reaches the test servers. The second question runs inside the locked production zone, the only place real chats live, and only the yes or no comes out.

NIST's guidance makes a related point for any de-identified data. Decide your goals first, then run re-identification studies to gauge the risk that people can be identified again.

Synthetic data's main limit is that it only contains what you and the generator thought of. Real users will always find something neither of you imagined.

• Blind spots. If no one lists voice-note transcripts as an input type, the grid never tests them.
• Style drift. Generated users tend to sound tidy and polite. Real ones type in fragments, with typos and mixed languages.
• Shared blind spots. If the generator and the bot run on similar models, they can share the same gaps, so the test misses them.
• Wrong rules. If the expected behaviour is wrong, the bot gets punished for being right. Policy experts must check the rules.

The team handles drift without exporting chats. Each month, the privacy officer's team runs a summary inside the locked zone. It counts which intents and languages real users sent, with no message text.

The QA team compares those counts with its grid. In September the counts showed 18 percent of real messages mixed Marathi and English (made up), while the grid had almost none. Adding Marathi-English typing as a persona surfaced four new failures.

Use pairwise coverage: choose rows so that every pair of values from any two dimensions appears together at least once. A full grid grows by multiplication, while a pairwise set grows far more slowly.

Say the team adds two dimensions of four values each: the language mix and the channel the message came from. Five dimensions of four values make 4 to the power 5, which is 1,024 rows before any repeats.

Sixteen is enough. We built a 16-row set for five dimensions and checked in code that it covers all 160 pairs. It is a standard design called an orthogonal array, where every pair of values appears in exactly one row. The smaller three-dimension case fits in a few lines, so you can see the trick.

import itertools

INTENTS = ["claim status", "cashless approval", "rejected claim", "add family member"]
PERSONAS = ["first-time user", "angry customer", "son of elderly parent", "Hinglish typist"]
TWISTS = ["none", "wrong policy number", "two claims at once", "asks about someone else"]

rows = [(INTENTS[i], PERSONAS[j], TWISTS[(i + j) % 4]) for i in range(4) for j in range(4)]

def pairs_covered(rows):
    return {(a, x, b, y) for r in rows
            for (a, x), (b, y) in itertools.combinations(enumerate(r), 2)}

full = list(itertools.product(INTENTS, PERSONAS, TWISTS))
print(len(rows), "rows cover", len(pairs_covered(rows)), "of", len(pairs_covered(full)), "pairs")

Run it with python3 and it prints one line. All 48 value pairs appear somewhere in the 16 rows:

16 rows cover 48 of 48 pairs

The trade-off is real. Pairwise keeps every pair but drops most three-way combinations. Our 16 rows hold only 16 of the 64 intent, persona and twist triples.

Why might pairs be enough? NIST's combinatorial testing research found most failures in the software it studied came from one or two parameters, and nearly all from six or fewer. Its ACTS tool generates such test sets.

Those studies cover conventional software, not LLM apps. So treat pairwise as a floor, and add full combinations by hand for the twists you fear most.

Simple regex filters miss formats you did not think of, and a near-copy check misses reworded duplicates. A regex, or regular expression, is a pattern that matches text. We ran both checks on Python 3.13.16 to see exactly where they fail.

Our first phone filter missed spaced numbers

Our first filter matched exactly 10 digits in a row, like 9876543210. In our test it let 98765 43210 and +91-98765-43210 straight through, two common ways to write one Indian mobile number.

The version below allows a space or a dash between digits. It flags any run of 10 or more digits, then any policy-like code that does not start with TEST.

import re
from difflib import SequenceMatcher

LONG_NUMBER = re.compile(r"\d(?:[ -]?\d){9,}")    # 10+ digits, even split by spaces or dashes
POLICY = re.compile(r"\b[A-Za-z]{2,5}-?\d{6,}\b")

def looks_real(text):
    if LONG_NUMBER.search(text):
        return True                                  # could be a phone or ID number
    return any(not p.startswith("TEST-") for p in POLICY.findall(text))

def near_copy(a, b):
    return SequenceMatcher(None, a, b).ratio() >= 0.9

def keep_case(case, kept_in_row):
    if looks_real(case["message"]) or not case.get("rules"):
        return False                                 # real-looking, or nothing to score
    return not any(near_copy(case["message"], k["message"]) for k in kept_in_row)

for text in ["Call me on 98765 43210 about TEST-482913",
             "Call me on +91-98765-43210",
             "Status of HLT-482913 please",
             "Status of TEST-482913 please"]:
    print(looks_real(text), "|", text)

Run it and it prints four lines. True means the message looks real, so the case is dropped:

True | Call me on 98765 43210 about TEST-482913
True | Call me on +91-98765-43210
True | Status of HLT-482913 please
False | Status of TEST-482913 please

What just happened? The first three messages look real, so they are dropped. Only the last one, with a policy number in the TEST format, survives.

The false alarm is cheap, since a dropped case is simply regenerated. The misses are the danger. Add a pattern for every format your real data uses, and keep the human review sample.

The Devanagari row is documented behaviour, not luck. Python's re module says that in text patterns, backslash d matches any Unicode decimal digit unless you pass the ASCII flag.

The near-copy check is slow and misses rewording

The near-copy check uses difflib's SequenceMatcher ratio, a similarity score from 0 to 1 where 1 means identical. A one-word change scored 0.98 in our test, so it was dropped. The same complaint, reworded, scored 0.64 and was kept.

So the check stops copy-paste repeats, not repeated ideas. Spotting the same idea in new words is a job for the human review sample.

It is also slow at scale. Comparing each new case with every kept case means up to 1,967,136 comparisons for 1,984 cases. On our test machine that projected to about 14 minutes, at 0.42 milliseconds per comparison.

Near-copies are most likely within one plan row, because the generator saw the same instructions there. Comparing only within the row cut the work to 27,350 comparisons, and the measured run took 9.3 seconds. Both timings used the same machine and our template-made test messages of about 117 characters.

Two more difflib details matter. Its docs call ratio expensive and offer quick_ratio and real_quick_ratio as cheaper upper bounds. The score can also depend on argument order, even for short texts. In our test, tide against diet scored 0.25, but diet against tide scored 0.5.

An automatic junk rule, which the docs call asymmetric, adds another cause once the second text reaches 200 characters. So pick one order and keep it: keep_case always passes the new case first.

A fixed seed does not make the test set repeatable

The seed fixes the plan's made-up numbers, not the generator's wording. Run the generator again and you get different chats, so this week's scores no longer compare fairly with last week's.

Save the generated set as a file with a version number, and rerun that file before each release. Regenerate only on purpose, such as when the grid changes.

Even the plan's numbers are only stable within one Python version. Python's documentation guarantees across versions only that random() repeats for the same seed, and says most other functions may change. Pin the version, or save the specs too.

Explain it as a pipeline with a privacy gate: plan, generate, filter, review, gate, run and score. Then name its limits and how you watch for drift.

1. 1

   State why real data is off limits

   Say the domain holds sensitive data, like health claims, so test systems must not hold real chats. Name the privacy owner who signs off.

2. 2

   Draw the coverage grid

   Pick dimensions such as intent, user type and twist, so rare situations get tested on purpose. If the full grid is too big, cover every pair of values instead.

3. 3

   Generate, then filter in code

   A model writes cases from the plan only. Code removes real-looking numbers, near-copies within each plan row and cases without expected rules.

4. 4

   Add review and a privacy gate

   People check a sample for realism and correct rules. A similarity check inside the locked zone blocks anything close to a real record.

5. 5

   Version the set and grow the hard cases

   Save each generated set with a version. Every failure becomes the starting point for harder variants, and passing cases stay as regression tests run before every release.

6. 6

   Name the limits

   Say synthetic chats are tidier than real ones and miss what nobody planned. Compare their mix with privacy-safe counts from production, and update the grid.