Prompt Versioning: Testing, Releases and Rollback

When you edit a live prompt in place, a one-line change can break behaviour you never touched, and nobody can say afterwards what the old text was. The prompt is part of the program, yet many teams change it with no review, no tests and no record. Prompt versioning fixes both problems.

Picture a common setup. A telemedicine team in Ahmedabad runs a chat assistant that books video consultations with doctors, in Gujarati, Hindi or English. It handles about 6,000 chats a day. The setup is made up, but the problem is real.

The assistant's instructions live in a system prompt, the hidden text the app sends to the model before every patient message. That prompt sat in one database row, editable from an admin page.

On a Friday evening, the clinical lead, the doctor who decides what the assistant may say, asked for shorter replies. The engineer on duty added one line: keep every reply under 40 words. To make room, they trimmed a long paragraph near the end, saved and went home.

By Monday, the clinical lead had found three kinds of trouble in the weekend chats.

• Eleven patients mentioned chest pain or trouble breathing. Before Friday, the assistant told such patients to call emergency services at once. After the trim, it offered them a consultation slot for Monday.
• While trimming, the engineer renamed a blank from {slot_time} to {time} by mistake. The app still supplied slot_time, and the admin page left unknown blanks untouched. Patients saw the raw text {time}.
• Some Gujarati replies switched to English halfway through. The tighter word limit was the likely cause, but nobody could check, because nothing recorded which text had produced which reply.

The first problem was dangerous. The clinical lead asked three plain questions: which prompt text answered these chats, what did it say before Friday, and who changed it? Nobody could answer quickly, because the old text had been overwritten in place.

The team rebuilt the old prompt from a screenshot in a team chat. That took three hours, while the risky version stayed live. The rest of this guide is the process that turns those three questions into a lookup.

Prompt versioning is the practice of saving every change to a prompt as a new, numbered version that never changes, and controlling which version each environment uses. An environment is one running copy of your app, such as staging or production. It gives prompts the same history, review, testing and rollback that ordinary code already has.

Think of a company that makes cough syrup. Every bottle carries a batch number, and a batch is never altered once it leaves the factory. When a complaint arrives, the maker looks up that batch and knows exactly what went into it. A prompt version ID does the same job for every reply your app sends.

Where the picture breaks: every bottle in a batch is the same. One prompt version can give different replies to the same message, because the model picks its words with some randomness. A version ID tells you exactly what text was sent, not exactly what came back.

Prompt versioning is not about writing better prompts. It is one part of running AI apps in production, and our guide on LLMOps covers the wider picture. Here the focus is narrow: changing prompts safely, whatever they say.

Five words used throughout this guide

Prompt template. Prompt text with blanks in it, which your code fills in for each request. Example: the template says the next free slot is {slot_time}, and the code fills in 4:30 pm.

Variable. One named blank in a template, like {slot_time} or {patient_language}. Your code must supply a value for every variable, or the request should fail.

Version ID. A label that points to one exact, frozen copy of a template. Example: booking-assistant v14 is the Friday-morning text, and v15 is the trimmed one.

Eval set. Short for evaluation set: a fixed list of test conversations with the behaviour you expect from each. Example: a patient who writes about chest pain must be told to call emergency services.

Canary. A small share of real users who get a new version first, so problems show up while few people are affected. Example: 5% of the clinic's patients get v16 for one day.

Keep the source of truth for your prompts in your Git repository, next to the code that fills them in. Add a prompt registry when you need to switch versions without a code deploy, which means shipping new code to your servers.

Git is a version control system: it records every change to your files, who made it and why. Work happens in a branch, a separate line of changes, and each saved change carries a commit message. A prompt kept as a file in Git gets history and code review for free.

A prompt registry is a service that stores each prompt version as a sealed record. It lets you point a label, such as production, at one of those versions. Your app asks the registry which version is live, so moving the label changes the live prompt without a deploy.

Many teams use both. Git holds the editable files and the review. When a change merges, a CI job publishes it to the registry as a new sealed version. CI, short for continuous integration, is the automatic system that builds and tests every change before it merges.

Here is where the clinic's prompts live now. Start at the prompts folder and follow the arrows to the model. Notice that nobody edits the registry by hand: only CI writes to it.

Test a prompt change in two layers. A person reviews the diff, and an automatic eval gate in CI runs the new version against fixed test conversations. If the gate fails, the change cannot merge.

A diff is the view that shows exactly which lines changed. For the Friday edit, a diff would have shown the trimmed emergency paragraph in red. The clinical lead would have spotted it in seconds.

Review is not enough on its own, because you cannot reliably predict how a model will react to new wording. So the team builds an eval set of 240 test conversations in three groups. Each group has a minimum pass rate the change must meet.

• Safety, 60 chats: urgent symptoms, questions about medicine doses, and requests the assistant must refuse. Every one must pass.
• Booking, 120 chats: finding slots, cancelling, rescheduling and fees. At least 95% must pass.
• Language, 60 chats: replies must stay in the patient's language. At least 90% must pass.

Here is the gate CI runs on every pull request, which is a proposed change waiting for review. The helpers load_cases and run_evals are stand-ins for code you would write. The run_evals helper sends each test chat through the given version and returns the share that passed, per group.

import sys

MIN_PASS = {"safety": 1.00, "booking": 0.95, "language": 0.90}

def eval_gate(version):
    cases = load_cases("evals/booking_assistant.jsonl")
    scores = run_evals(version, cases)    # share passed, per group
    failed = [g for g, bar in MIN_PASS.items() if scores[g] < bar]
    for group in failed:
        print(f"{group}: {scores[group]:.2f} is below {MIN_PASS[group]:.2f}")
    return not failed

if __name__ == "__main__":
    sys.exit(0 if eval_gate(sys.argv[1]) else 1)

Here is what the gate does. MIN_PASS sets a bar per group, and safety must be perfect, so one failed emergency case blocks the merge. The last line exits with 1 on failure, which is how CI knows to stop the change.

You can check it with fake helpers. Say run_evals returns 59 of 60 for safety and 0.88 for language. The gate prints two lines, safety: 0.98 is below 1.00 and language: 0.88 is below 0.90, then exits with 1.

One trap: a model's replies vary from run to run, so a case that passed once can fail tomorrow. Run each safety case several times, say five, and count it as passed only if every run passes.

Teams often skip even the record of what changed. A study presented at MSR 2025, a software engineering research conference, looked at 1,262 prompt changes across 243 GitHub repositories. Only 21.9% of those changes were documented in commit messages.

The same study found that prompt changes can introduce logical inconsistencies, and its authors call for testing frameworks and automated validation tools built for prompts. For how to choose and score test cases, see our guide on LLM evaluation metrics and test frameworks.

Roll out in stages: first a staging environment with no real patients, then a canary of a few percent of real users, then everyone. At each stage, compare the new version with the old one and stop at the first bad sign.

The Twelve-Factor App, a well-known set of rules for web apps, defines config as everything likely to vary between deploys. It requires strict separation of config from code.

For prompts, the split is clean. The text is code and lives in Git, while the choice of which version runs where is config. Each environment reads a small release config like this one.

{
  "prompt": "booking-assistant",
  "stable_version": "v15",
  "canary_version": "v16",
  "canary_percent": 5
}

A canary release sends a small group of users to the new version first, then widens as confidence grows. If problems appear, users are routed back to the old version. The clinic's plan for v16 has four stages.

1. 1

   Staging for one day

   v16 runs in staging against the eval set and a replay of 500 anonymised past chats. The engineer and the clinical lead read a sample by hand.

2. 2

   Canary at 5%

   5% of patients get v16 for 24 hours. The same patient always gets the same version, so a conversation never switches halfway.

3. 3

   Compare with v15

   The team compares four numbers between the two groups: urgent-symptom handoffs, completed bookings, reply length and requests for a human.

4. 4

   Widen to 25%, then 100%

   If nothing is worse, the share grows. If a stop rule fires, the canary drops to 0% and v15 serves everyone again.

The clinic's stop rule is short. Any fall in urgent-symptom handoffs, or completed bookings down by more than 2 points, ends the canary at once.

Know what a canary cannot see. Five percent of 6,000 chats is about 300 chats a day, so chest-pain cases may show up only a few times. Rare dangers must be caught by the gate; the canary is the backstop.

In the diagram, notice that a version must pass two checks before everyone gets it, and that a failed canary leads to a config change, not a deploy.

You roll back by pointing the release config at the previous version, which is already sealed and tested. Nothing is retyped, rebuilt or redeployed. In the clinic's setup, the app re-reads its config every 30 seconds, so a rollback reaches every server within about a minute (made up).

Go back to the syrup maker. When a batch turns out bad, the maker recalls that batch number, and shops go back to selling the previous batch. Nobody tries to fix bottles on the shelf. A prompt rollback works the same way: v16 is pulled, and v15 serves again.

In the config above, rollback is one edit: set canary_percent to 0, or set stable_version back to the last good number. Pete Hodgson's article on feature toggles calls this kind of switch an ops toggle, which operators must be able to flip very quickly.

Rollback only stays this easy if the old version still works with today's code. So for a change that renames or adds variables, the team first ships code that can fill both the old and the new template. Only then does the new prompt go out.

These habits keep the clinic's rollbacks fast:

• The previous version is sealed in the registry, not rebuilt from memory.
• The engineer on call (the one who answers alerts) that week may roll back without asking anyone. Rolling forward always needs review.
• A stop rule can trigger the rollback automatically during a canary.

Give every prompt a stable name and every change a new number that is never reused, such as booking-assistant v15. Treat the template's variables as part of the version, because a renamed variable breaks the code that fills it.

Semantic Versioning 2.0.0, a widely used spec for numbering software releases, has a rule that fits prompts exactly. Once a version is released, its contents must not be modified. Any change must be released as a new version.

Its three-part numbers can work for prompts too, if you decide what counts as breaking. The clinic borrows the idea like this:

• Major, like 2.0.0: the variables or the reply format change, so the app code must change too.
• Minor, like 1.4.0: behaviour changes, such as a new rule about cancellations, but the code that fills the template stays the same.
• Patch, like 1.4.1: a wording fix that should not change behaviour. Still run the eval gate, because the model may disagree.

Plain counters like v14 and v15 work fine too. What matters is that one number always means one text. Some teams also store a hash, a short fingerprint computed from the text, so a log line can prove which words were sent.

A version should also record the model and settings it was tested with. The same prompt can behave differently on another model, or at another temperature (a setting that controls how random the replies are). If your provider lets you pin an exact model snapshot, a dated release of the model that does not change, pin it.

Make missing variables fail loudly

The {time} bug happened because the admin page left unknown blanks in place without complaint. In Python 3.13, both str.format and the substitute method of string.Template raise a KeyError, Python's error for a missing name, when a blank has no value.

Watch for the quiet versions. string.Template's safe_substitute does exactly what the admin page did: it returns the text with $time still in it. Extra values are ignored silently too, so code that still passes a value nobody uses gets no warning.

The eval gate also runs a fill check on every changed template. This one compares the blanks in two versions and then tries the new one. Run it and it prints ['slot_time', 'time'], then the last line raises KeyError: 'time'.

from string import Formatter

def fields(template):
    return {name for _, name, _, _ in Formatter().parse(template) if name}

old = "Your slot is at {slot_time}. Reply in {patient_language}."
new = "Your slot is at {time}. Reply in {patient_language}."
print(sorted(fields(old) ^ fields(new)))
new.format(slot_time="4:30 pm", patient_language="Gujarati")

JSON examples break str.format

Prompts often show the model a JSON example, like {"slot": "4:30 pm"}. To str.format, those braces look like a blank, so Python 3.13 raises KeyError: '"slot"' before any request is sent.

Two fixes work. Double the literal braces, writing {{ and }}, as the Python docs describe. Or use string.Template, whose blanks start with a dollar sign, so JSON braces pass through untouched. Both were run on Python 3.13.16.

One more rule: patient text never goes into a blank inside the instructions. It goes in the user message, which keeps your rules and the patient's words apart. That lowers the risk of a patient rewriting the rules, but does not remove it.

Hash each patient's ID into one of 100 buckets, and send buckets below canary_percent to the new version. A hash function turns any text into a fixed-size fingerprint, and the same input always gives the same output, so a patient never flips versions mid-chat.

If every rollout hashes only the patient ID, the next canary gets the same patients. In a test with 1 lakh (100,000) made-up IDs, all 4,873 patients in the v16 canary were also in the v17 canary.

That puts every new risk on the same few patients, and any quirk of that group skews every canary. The fix is a salt, extra text mixed into the hash. Salted with the version name, the v16 and v17 canaries shared only 264 of 5,127 patients, close to the 5% you would expect by chance.

import hashlib

def pick_version(patient_id, config):
    key = f"{config['canary_version']}:{patient_id}"    # fresh salt per rollout
    digest = hashlib.sha256(key.encode()).hexdigest()
    bucket = int(digest, 16) % 100                      # same patient, same bucket
    if bucket < config["canary_percent"]:
        return config["canary_version"]
    return config["stable_version"]

config = {"stable_version": "v15", "canary_version": "v16", "canary_percent": 5}
on_canary = sum(pick_version(f"patient-{n}", config) == "v16" for n in range(100_000))
print(on_canary, "of 100000 patients on v16")

Run it and it prints 5127 of 100000 patients on v16, which is 5.13%. At canary_percent 25 the same test put 24,996 patients on v16. Every patient from the 5% stage stayed on v16 as the share grew.

Keep the salt fixed for the whole rollout. If you changed it between 5% and 25%, patients would be reshuffled, and some would switch versions halfway through a conversation.

Log the prompt version with every request, so any reply can be traced to the exact text that produced it. Without it, a complaint turns into guesswork about which prompt was live at that moment.

This is the batch number again. A complaint about one bottle is useful only if the bottle shows its batch. Each log line now carries the prompt name, version, model, temperature, token counts and how long the reply took. Tokens are the small pieces of text that model services count and charge by.

OpenTelemetry, an open-source standard for traces (records of each request's steps), metrics and logs, publishes naming conventions for generative AI calls. Its attribute list includes gen_ai.prompt.name and gen_ai.prompt.version, next to gen_ai.request.model and gen_ai.usage.input_tokens.

As of 19 October 2026, all four are marked Development, so names may still change; pin the conventions version you follow. The list says the version string can use any scheme, and should match your prompt management system's version ID.

Be careful with message text. The conventions say instrumentation, the code that records this monitoring data, should not record instructions, inputs and outputs by default, and should offer an opt-in. The clinic logs versions and numbers by default, and keeps chat text only in a locked store for a limited time.

Here is one chat, traced from start to finish. Each box is one step, and the detail line shows who talks to whom.

Each prompt needs one named owner who approves changes, plus clear rules for who else must sign off on which parts. Engineers own the template and the code around it. Domain experts own the lines that carry their rules.

At the clinic, the engineer owns the booking-assistant prompt. The clinical lead owns its safety section: urgent symptoms, medicine questions and anything that sounds like a diagnosis. So the safety rules move into their own file.

On GitHub, a CODEOWNERS file names the owners of a file or folder, and they are asked to review any pull request that touches it. A branch protection rule can then require their approval before merging. As of October 2026, code owners are not automatically asked to review draft pull requests, and code owners in private repositories need a paid GitHub plan.

The new process fits in four rules:

• Anyone may propose a change, in a branch, with a one-line reason in the commit message.
• The prompt's owner reviews every change, and the clinical lead must also approve any change to the safety file.
• The eval gate must pass, and nobody can skip it, including the owner.
• Only the release config can make a version live, and every change to it is logged with a name and a time.

Explain it as normal release engineering applied to prompts: version, test, release in stages, observe and roll back. Then add the parts that are special to models, such as pinning the model and settings, and repeating eval runs.

1. 1

   Treat the prompt as code

   Say where prompts live: templates in Git, sealed versions in a registry, and a release config per environment.

2. 2

   State the version rule

   A published version never changes. Every fix is a new version, and each one records its model and settings.

3. 3

   Put a gate in CI

   Describe an eval set with groups and pass bars, where every safety case must pass on repeated runs.

4. 4

   Roll out with a canary

   Give the stages, the share of users, the numbers you compare and the stop rule. Hash users into buckets, with a fresh salt per rollout.

5. 5

   Make rollback a pointer change

   Show that rollback is one config change, and that old versions stay compatible with current code.

6. 6

   Close with logging

   Every request logs the prompt name and version, so any reply can be traced. Mention privacy when the data is sensitive.