LLM Model Deprecation: How to Migrate Without Breaking Users

Your feature keeps working until the shutdown date, and then every call to that model fails. Worse, a rushed switch to the replacement often breaks things quietly, long before the date.

LLM model deprecation is the notice that starts this clock. A hosted model is one you call over the internet from a model provider, a company that runs the model and charges per use. You do not control when it changes.

Picture a common setup. An online grocery app in Pune uses a hosted model to triage about 40,000 support tickets a day. The setup and its numbers are made up, but the failure pattern is real.

Each ticket goes to the model with a fixed prompt. The model must return a small JSON object with two keys: a category such as refund or late_delivery, and a short reply. The app routes the ticket using the category.

On 23 October 2026 the team gets a notice. Their pinned model retires on 15 January 2027, 84 days away. The provider suggests a newer model and says it is better on most tasks.

An engineer swaps the model ID on a Friday. By Monday, some tickets have landed in the wrong queue, because a few replies came back wrapped in extra text that the JSON parser rejected. Nobody changed the prompt.

This guide follows that triage feature through a safe migration. You will see where the model ID was hiding, how to test the new model on real traffic, and how to switch back if it misbehaves.

You need six words: deprecation, retirement, model ID, alias, golden set and rollback. Shadow and canary come later, in their own section.

Deprecation. Deprecation means the provider has announced a model is being phased out. It usually still works for now. RFC 9745, the HTTP standard for the Deprecation header, says deprecation itself does not change how a resource behaves.

Retirement. Retirement, also called shutdown or sunset, is the date calls to the model stop working. Between the deprecation notice and retirement is your migration window. In the running example it is 84 days.

Model ID and alias. The model ID is the exact string your code sends to pick a model. A dated ID names one fixed release, such as support-model-2026-03-01 in our made-up example. An alias is a short name like support-model-latest that the provider can point at newer releases.

Golden set. A golden set is a fixed list of real past requests with answers your team agrees are right. You run every candidate model on it, so you compare models on the same questions.

Rollback. Rollback means switching back to the old model fast when the new one misbehaves. It should be one config change, not a code deploy.

LLM model deprecation is the planned end of life of one model version. It breaks apps because the replacement behaves differently, even when it scores better overall. Your prompts were tuned to the old model's habits.

Research shows how large these differences can be. One study compared the March and June 2023 versions of two hosted models. For one model, accuracy on a prime-number task fell from 84 percent to 51 percent.

The same study found that the later versions made more formatting mistakes when generating code, and that one model followed user instructions less reliably. The authors concluded that hosted models need continuous monitoring.

A second study found that tiny prompt formatting changes, such as spacing or separators, moved accuracy by up to 76 points on one openly released model. That sensitivity did not go away with bigger models, more examples in the prompt, or extra training to follow instructions.

Put the two findings together. A new model can react to your exact prompt very differently from the old one. A prompt that worked for a year is a set of habits tuned to one model.

Model IDs hide in more places than the main code file. The triage team found them in several places, and only one was the obvious one.

Search for the word model across code and config files first. This command lists every matching line, with file names and line numbers, in the current folder and below. It is noisy on purpose, so read the hits rather than counting them. Add an --include pattern for every file type your stack uses, such as .toml, .ts or .tf files.

grep -rniE "model" --include="*.py" --include="*.yaml" --include="*.yml" --include="*.json" --include="*.env" .

On a small test folder built for this guide, the command found five matching lines. They included a dated ID in Python code and an alias in a deployment values file. They also found a fallback model, a backup the app calls when the main model fails, that retires even sooner.

A text search only covers files. Model IDs also live where grep cannot reach. Check each place in the table below and write every hit into one shared list.

Your traces are the final check. A trace span is one timed record of a call, saved by your monitoring tool. OpenTelemetry, an open standard for traces and metrics, names the requested model gen_ai.request.model.

Group a week of spans by that attribute. Any model ID you see there is in use somewhere, even if your text search missed it.

Pin a dated model version in production. An alias can move to a new release without any change on your side, so behaviour can shift on a day you did not choose.

Software has a long rule for this. Semantic Versioning, a widely used spec for version numbers, says a released version's contents must not be modified. Any change must ship as a new version.

A dated model ID is the closest thing you get to that promise. Model providers set their own versioning rules, so read your provider's policy for what a dated ID guarantees.

Pause and predict. If your code asks for a dated ID, could the response still come from a different model? You cannot rule it out without checking, so log what actually answered.

OpenTelemetry has a second attribute for this, gen_ai.response.model, described as the name of the model that generated the response. Its documented examples show a short request name and a dated response name.

This short script reads exported spans and counts which model was asked for and which one answered. It flags any call that the pinned model did not serve.

from collections import Counter

PINNED = "support-model-2026-03-01"  # made-up dated model ID

# Exported trace spans, using OpenTelemetry GenAI attribute names.
spans = [
    {"gen_ai.request.model": "support-model-2026-03-01", "gen_ai.response.model": "support-model-2026-03-01"},
    {"gen_ai.request.model": "support-model-latest", "gen_ai.response.model": "support-model-2026-09-15"},
    {"gen_ai.request.model": "support-model-2026-03-01", "gen_ai.response.model": "support-model-2026-03-01"},
]

requested = Counter(s["gen_ai.request.model"] for s in spans)
served = Counter(s["gen_ai.response.model"] for s in spans)
print("requested:", dict(requested))
print("served:   ", dict(served))
strays = [s for s in spans if s["gen_ai.response.model"] != PINNED]
print("calls not served by the pinned model:", len(strays))

requested: {'support-model-2026-03-01': 2, 'support-model-latest': 1}
served:    {'support-model-2026-03-01': 2, 'support-model-2026-09-15': 1}
calls not served by the pinned model: 1

What just happened? One call used the alias, and the alias was quietly served by a newer release. The script caught it because it trusts the response attribute, not the request.

Sample real requests from your logs, remove personal data, and have people agree on the right answer for each. Real traffic matters because it holds the odd inputs your users actually send.

An eval set is any fixed list of test inputs with expected results. It becomes golden when the team trusts the expected answers enough to block a release on them.

• Sample by category, not just at random. The triage team took cases from every ticket category, so rare ones such as address changes were present.
• Add the cases that broke before. Every past incident ticket becomes a permanent test case.
• Add the edge cases you fear: very long tickets, mixed Hindi and English text, angry messages, and tickets with no clear category.
• Remove personal data before storing. Replace phone numbers, addresses and order IDs with fake values of the same shape.
• Record the agreed answer and who agreed. When two reviewers disagree, the case is ambiguous and should be marked so.

Size depends on how many kinds of input you have. Our rule of thumb is 100 to 300 cases to start, covering every category. This is a starting habit, not a standard.

Also save the old model's outputs for every case, with latency and token counts. Once the old model retires, those saved outputs are your only record of how it behaved.

Compare six things: output format, refusals, correctness per case, length, latency and cost. A single accuracy score hides most migration bugs.

Two terms first. A token is a small piece of text, roughly a word or part of one, and providers bill per token. Latency is the time from sending a request to receiving the full answer.

p95 latency is the time that 95 percent of calls beat. It shows the slow tail better than an average does. With only a handful of cases, p95 is close to the slowest call, so use hundreds of calls before trusting it.

Why accuracy alone hides regressions

A negative flip is a case the old model got right and the new model gets wrong. A paper on image classifiers defines negative flips this way. Later work found the same problem when language models are updated.

Users notice negative flips even when the new model wins overall. A customer whose refund ticket always worked now gets a wrong answer. A better average does not help that customer.

A small diff harness you can run

The script below uses saved, made-up outputs from both models for six golden cases. It needs only the Python standard library and makes no network calls. Prices are made up.

import json
import operator
import re
import statistics

# Each golden case: a real-looking ticket, the label a human agreed on,
# and the saved output, latency and token counts from the old and new model.
CASES = [
    {"id": "t01", "expected": "refund",
     "old": {"text": '{"category": "refund", "reply": "Refund started."}', "ms": 820, "in": 410, "out": 30},
     "new": {"text": '{"category": "refund", "reply": "Your refund has been started and should reach you in 5-7 days."}', "ms": 640, "in": 410, "out": 52}},
    {"id": "t02", "expected": "late_delivery",
     "old": {"text": '{"category": "late_delivery", "reply": "Sorry for the delay."}', "ms": 790, "in": 395, "out": 28},
     "new": {"text": '```json\n{"category": "late_delivery", "reply": "Sorry for the delay."}\n```', "ms": 610, "in": 395, "out": 34}},
    {"id": "t03", "expected": "address_change",
     "old": {"text": '{"category": "address_change", "reply": "Address updated."}', "ms": 860, "in": 402, "out": 27},
     "new": {"text": '{"category": "address_change", "reply": "Address updated."}', "ms": 655, "in": 402, "out": 27}},
    {"id": "t04", "expected": "refund",
     "old": {"text": '{"category": "refund", "reply": "We will check the charge."}', "ms": 900, "in": 430, "out": 31},
     "new": {"text": "I'm sorry, but I can't help with payment disputes.", "ms": 580, "in": 430, "out": 14}},
    {"id": "t05", "expected": "other",
     "old": {"text": '{"category": "refund", "reply": "Refund started."}', "ms": 840, "in": 388, "out": 30},
     "new": {"text": '{"category": "other", "reply": "Thanks, we have passed this on."}', "ms": 600, "in": 388, "out": 33}},
    {"id": "t06", "expected": "late_delivery",
     "old": {"text": '{"category": "late_delivery", "reply": "Your order is on the way."}', "ms": 810, "in": 401, "out": 29},
     "new": {"text": '{"category": "address_change", "reply": "Please confirm your address."}', "ms": 630, "in": 401, "out": 31}},
]

REQUIRED_KEYS = {"category", "reply"}
REFUSAL = re.compile(r"\b(i can't|i cannot|i'm unable|i am unable)\b", re.IGNORECASE)
PRICE_PER_MILLION = {"old": {"in": 40.0, "out": 160.0}, "new": {"in": 30.0, "out": 120.0}}  # rupees, made up


def parse(text):
    """Return the parsed dict if the text is exactly a JSON object with the keys, else None."""
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        return None
    if not isinstance(data, dict) or set(data) != REQUIRED_KEYS:
        return None
    return data


def p95(values):
    # Simple index-based p95: fine for a quick check, use a stats library for reports.
    ordered = sorted(values)
    return ordered[int(0.95 * (len(ordered) - 1))]


def summarise(side):
    rows = [c[side] for c in CASES]
    parsed = [parse(r["text"]) for r in rows]
    valid = [p for p in parsed if p is not None]
    correct = [c["id"] for c, p in zip(CASES, parsed) if p is not None and p["category"] == c["expected"]]
    price = PRICE_PER_MILLION[side]
    cost = sum(r["in"] * price["in"] + r["out"] * price["out"] for r in rows) / 1_000_000
    return {
        "format_ok": round(len(valid) / len(rows), 2),
        "refusals": sum(1 for r in rows if REFUSAL.search(r["text"])),
        "accuracy": round(len(correct) / len(rows), 2),
        "correct_ids": set(correct),
        "median_out_tokens": statistics.median(r["out"] for r in rows),
        "p95_ms": p95([r["ms"] for r in rows]),
        "cost_per_1000_rupees": round(cost / len(rows) * 1000, 2),
    }


old, new = summarise("old"), summarise("new")
for key in ["format_ok", "refusals", "accuracy", "median_out_tokens", "p95_ms", "cost_per_1000_rupees"]:
    print(f"{key:22} old={old[key]!s:8} new={new[key]!s}")

negative_flips = sorted(old["correct_ids"] - new["correct_ids"])
positive_flips = sorted(new["correct_ids"] - old["correct_ids"])
print("negative flips (old right, new wrong):", negative_flips)
print("positive flips (old wrong, new right):", positive_flips)

# Gate: block the switch if the new model breaks format or refuses more, whatever its accuracy.
blockers = []
if operator.gt(old["format_ok"], new["format_ok"]):
    blockers.append("format got worse")
if operator.gt(new["refusals"], old["refusals"]):
    blockers.append("more refusals")
if negative_flips:
    blockers.append(f"{len(negative_flips)} negative flips to read by hand")
print("verdict:", "BLOCK: " + "; ".join(blockers) if blockers else "PASS")

Running it on Python 3.13 prints this. Your numbers will match if you copy the cases exactly.

format_ok              old=1.0      new=0.67
refusals               old=0        new=1
accuracy               old=0.83     new=0.5
median_out_tokens      old=29.5     new=32.0
p95_ms                 old=860      new=640
cost_per_1000_rupees   old=20.84    new=15.95
negative flips (old right, new wrong): ['t02', 't04', 't06']
positive flips (old wrong, new right): ['t05']
verdict: BLOCK: format got worse; more refusals; 3 negative flips to read by hand

What just happened? The new model is faster and about 23 percent cheaper, yet the gate blocks it. Case t02 came back wrapped in a code fence, the three backticks that mark code in chat-style text. The JSON parser rejected it.

Case t04 was refused outright, and t06 picked the wrong queue. The new model did fix t05, a positive flip. Positive flips are good news, but they never cancel a negative flip.

The refusal check is a rough keyword match. It misses refusals worded another way, so read flagged and unparsed replies by hand.

At 40,000 tickets a day, the made-up prices give about Rs 834 a day for the old model and Rs 638 for the new one. That saving is worthless if refund tickets go to the wrong queue.

In shadow testing, each live request also goes to the new model in the background, and only the old model's answer reaches the user. In a canary release, a small share of users then get the new model for real.

The new model's answers are stored for comparison and never used. You test on real traffic while users still see only the old answers.

Send the shadow call asynchronously, so a slow new model cannot delay the live answer. Shadow calls must never take actions. If your feature sends emails, issues refunds or calls tools, the shadow path must only record what it would have done.

Shadowing all traffic means two model calls per request, so you pay for both while it runs. You can shadow a sample, such as 10 percent of traffic, to keep the cost down.

A canary release, in the classic definition, sends a small group of users to the new version first. The rollback is to route those users back to the old version until the problem is fixed.

1. 1

   Offline gate

   Run the diff harness on the golden set. Fix every format break and read every negative flip before going further.

2. 2

   Shadow for a few days

   Send a sample of live tickets to the new model in the background. Compare format, categories, latency and token counts on the dashboard.

3. 3

   Canary at 1 percent

   Route 1 percent of tickets to the new model for real. Watch misrouted tickets, customer complaints and timeout errors.

4. 4

   Widen in stages

   Move to 10, then 50, then 100 percent, pausing at each stage. These stage sizes are an example, not a rule.

5. 5

   Keep the old ID ready

   Leave the old dated ID in config as the rollback target until the full switch is stable.

Change one thing at a time and rerun the golden set after each change. Treat the prompt and the model ID as one release, versioned together.

In the running example, the fix for case t02 was one extra instruction: reply with the JSON object only, with no code fence. The team kept a parser fallback that strips fences too.

The refusal in t04 needed a different fix. The prompt now says that billing questions are in scope and should be routed to the refund queue. The team retested every refund case afterwards.

Do not tune the prompt on the golden set alone. If you change the prompt until every golden case passes, you may have fitted the test. Keep a second, smaller hold-out set that you only check at the end.

Format sensitivity is the reason for this care. A wording change that fixes one case can move other cases, so the whole set must run after every edit.

Keep the model ID in config, behind a flag (a setting you can change without redeploying), so one change sends traffic back to the old dated ID. A rollback that needs a code deploy is too slow for a live incident.

Here is the catch that is easy to miss. Rollback to the old model only works until its retirement date. After that date, the old ID fails, so your rollback target must be some other model.

Plan for that early. Before the old model retires, test a second fallback model on the golden set too. In the running example, the team had to drop its listed fallback, because it retired even sooner.

Keep one small file that lists every model ID you use, where it is used, and its announced retirement date. Review it on a schedule and alert well before each date.

Two RFCs help when an API sends their headers. An RFC is a numbered document from the IETF, the group that designs internet protocols such as HTTP. The Deprecation header, RFC 9745 from March 2025, says a resource is or will be deprecated. The Sunset header, RFC 8594, gives the date it may stop responding.

RFC 9745 also says the Sunset date must not be earlier than the Deprecation date. Not every API sends these headers, so treat them as a bonus signal, not your only source.

from datetime import date
from email.utils import parsedate_to_datetime

TODAY = date(2026, 10, 23)

# Where each pinned model ID is used, and its announced retirement date (all made up).
calendar = {
    "support-model-2026-03-01": {"used_by": "ticket triage", "retires": date(2027, 1, 15)},
    "summary-model-2025-11-20": {"used_by": "weekly digest, triage fallback", "retires": date(2026, 12, 1)},
    "embed-model-v2": {"used_by": "search index", "retires": None},
}

# If an API sends an RFC 8594 Sunset header, parse it instead of copying dates by hand.
sunset_header = "Fri, 15 Jan 2027 00:00:00 GMT"
print("Sunset header date:", parsedate_to_datetime(sunset_header).date())

rows = sorted(
    ((info["retires"] - TODAY).days, model, info["used_by"])
    for model, info in calendar.items() if info["retires"]
)
for days_left, model, used_by in rows:
    print(f"{days_left:4d} days left  {model}  ({used_by})")
print("no date announced:", [m for m, i in calendar.items() if not i["retires"]])

Sunset header date: 2027-01-15
  39 days left  summary-model-2025-11-20  (weekly digest, triage fallback)
  84 days left  support-model-2026-03-01  (ticket triage)
no date announced: ['embed-model-v2']

Sort by days left and the urgent work rises to the top. The weekly digest model has only 39 days, so it should be migrated before the triage model, even though triage handles more traffic.

Risk frameworks ask for this kind of plan too. The NIST AI Risk Management Framework 1.0 asks for contingency processes to handle failures or incidents in third-party AI systems deemed high-risk.

Frame it as a forced dependency upgrade with a hard deadline. Then walk through detection, testing, release and rollback, naming a metric at every stage.

1. 1

   State the risk

   A hosted model will stop working on a set date, and its replacement behaves differently. The goal is to migrate with no user-visible regression.

2. 2

   Inventory and pin

   Find every model ID in code, config, routers and eval jobs. Pin dated IDs and log the model that actually answered.

3. 3

   Build the golden set

   Sample real traffic by category, remove personal data, agree expected answers and save the old model's outputs.

4. 4

   Gate offline

   Compare format, refusals, negative flips, length, p95 latency and cost. Block on format breaks and unread negative flips.

5. 5

   Shadow, then canary

   Shadow a sample of traffic with no side effects, then canary at a small percentage and widen in stages.

6. 6

   Plan rollback with an expiry

   Rollback is a config flip to the old ID until it retires. After that, a tested second model is the fallback.

Expect one follow-up question: how do you know the new model is better for your users? Answer with your golden set and negative flips, not the provider's general claims.