LLMOps Explained: From MLOps to Production LLMs

Its answers change, and nobody can say why. An LLM app's answers depend on its prompt, its model, the model's settings and the data it searches, not just its code. LLMOps is the set of habits that makes each of those changes visible, tested and reversible.

Picture a common setup. A cloud kitchen, a business that cooks only for delivery, runs a menu assistant inside its ordering app. The setup is made up, but the problem is real.

Customers ask whether a dish has peanuts, how spicy a curry is, or which bowls are Jain-friendly (no onion, garlic or root vegetables). The assistant answers about 6,000 questions a day.

It has three parts. A prompt, the instructions sent to the model with every question, tells it to answer only from the menu. A retrieval index, a searchable store of menu and recipe cards, supplies the facts. A large language model, or LLM, reads both and writes the reply.

For five weeks it works well. Then three things change in one week. The team shortens the prompt so replies fit a phone screen. The menu team adds 12 dishes and rebuilds the index. The model service moves its default model name to a newer version.

On Saturday, a customer asks whether the new Thai curry bowl contains peanuts. The bot says no. The recipe card says yes, and the customer spots the peanuts before eating. The head of engineering asks which change caused it.

Nobody can say. None of the three changes was recorded with a date, and nothing tested the bot before or after them. The rest of this guide builds the habits that turn that guess into a five-minute lookup.

Six words carry most of this guide. Each one is used in the same sense from here to the end.

• Model service: a model someone else hosts, which you call over the internet and pay for per use. You do not control when its default model changes.
• Release: one exact combination of everything that shapes an answer: prompt, model version, settings, retrieval index and code, plus the eval set it was tested on.
• Eval set: short for evaluation set, a fixed list of test questions, each with what a good answer must contain. Example: the Thai curry bowl peanut question must get yes.
• Environment: a separate copy of the app for one purpose. Development is for trying ideas, staging is a private copy that matches production, and production is what customers use.
• Canary: a release that gets a small share of real traffic, such as 5 percent, before everyone else. If it misbehaves, only a few customers see it.
• Rollback: switching back to the last release that worked. With named releases, it is usually one config change (editing a setting, not code).

LLMOps is the set of practices, tools and team habits for running LLM apps in production. It covers how you change, test, ship, watch and improve them, with the same discipline you already give code.

The name copies MLOps. A 2022 paper on MLOps defines it as including best practices, sets of concepts and a development culture. The same paper calls MLOps a vague term, so expect definitions of LLMOps to vary too.

The idea underneath is simple. An LLM app is software whose behaviour lives partly outside its code. LLMOps gives those outside parts what code already gets: review, tests, versions and a way back.

MLOps is mostly about training and serving models you build from your own data, while LLMOps is mostly about steering a model someone else trained. The goals match, but the parts you change are different.

Think of two farmers, and map each part to the app.

• The farmer who breeds her own seed is an MLOps team. It trains its own model on its own data, so it controls the plant itself.
• The farmer who buys seed from a seed company is an LLM team calling a model service. It cannot change the seed.
• Soil, water and the week of sowing are the prompt, the retrieval data and the settings. These are the parts the LLM team controls.
• The seed company quietly changing its seed is the model service moving a model name to a new version. That is what hit the menu assistant.

Where the picture breaks: a farmer learns about bad seed only at harvest. You can rerun your eval set on the same day the model changes, but only if you already have one.

So treat the model as a dependency, like a library you import. Unlike a pinned library, a model called by a moving name can change under you, in ways no test of yours has seen.

The table compares the two for a team that calls a model service. Tokens, mentioned in it, are small pieces of text, often a word or part of one. As of October 2026, many model services charge per token, with separate rates for tokens in and tokens out.

Plenty carries over unchanged. A 2022 interview study of 18 ML engineers found three variables that govern success in production: velocity, validation and versioning. Velocity is how fast you can try ideas. Validation is testing changes and catching bugs as early as possible. Versioning is keeping several versions of models and datasets, so you can look up, debug and switch back.

An older warning carries over too. A 2015 paper on hidden technical debt in ML systems listed risk factors including configuration issues and changes in the external world. In an LLM app, the prompt and settings are configuration, and the model service is part of the external world.

A useful way to draw the LLMOps lifecycle is six stages that run in a loop: data, prompts, evaluation, deployment, monitoring and feedback. What users do in production feeds the next round of data and prompts.

• Data: prepare what the model reads when it answers. For the menu assistant, each recipe card carries a date and an owner, and a new dish enters the index only after its allergens are filled in.
• Prompts and model choice: write the instructions and pick the model and its settings, such as temperature (how much randomness the model adds). Keep prompts in files, reviewed like code.
• Evaluation: run the eval set against a new release and compare its scores with the live one. The menu assistant starts with 80 questions, 25 of them about allergens.
• Deployment: move a release from staging to production, ideally through a canary. Because a release is mostly names and settings, this often means changing one config value.
• Monitoring: watch quality signals, cost per answer, reply time and errors. The hard part is silent regressions, where quality drops and nothing throws an error.
• Feedback: turn a thumbs-down, a complaint or a support correction into a new test question for the next round.

Expect the eval set itself to change. A 2024 study of an evaluation tool watched nine industry practitioners grade LLM outputs. Grading outputs helped them define their criteria, so their criteria kept shifting. The authors call this criteria drift, and it is why the eval set needs a version too.

Several of these stages have whole guides of their own. See our guides on prompt versioning, testing and rollback, on LLM evaluation metrics and test frameworks, and on LLM observability.

Version everything that can change an answer: the prompt, the model version and settings, the retrieval index, the eval set and the code. Then tie them together under one release id that is saved with every answer.

• Prompt: the exact text, stored in a file with a version in its name, such as menu_answer_v14. A one-word edit can change tone, length or safety.
• Model version and settings: the exact version name your service gives, not an alias such as latest, plus temperature and length limits.
• Retrieval index: when it was built, which documents are in it, and how they were split into chunks (small pieces for search). A rebuilt index is a new release, even with the same prompt.
• Eval set: its version, so you know which questions a score came from. A 90 percent score on 80 questions and on 120 questions are different results.
• Code and tools: the usual way, in Git, the version control system most teams already use.

The code below writes one release as a small dictionary and turns it into a short id. It uses SHA-256, a hash: it turns any text into a fixed-length fingerprint, and the same text always gives the same fingerprint. It needs nothing beyond the standard library, and we ran it on Python 3.11, 3.12 and 3.13. Your app then saves that id next to every answer it logs.

import hashlib
import json

release = {
    "prompt": "menu_answer_v14",
    "model": "large-model-2026-08-12",
    "temperature": 0.2,
    "index": "menu_index_2026-10-09",
    "eval_set": "menu_evals_v6",
    "code": "commit-4f9a2c1",
}

def release_id(bundle):
    text = json.dumps(bundle, sort_keys=True)
    return hashlib.sha256(text.encode()).hexdigest()[:10]

print(release_id(release))

shuffled = dict(reversed(list(release.items())))
print(release_id(shuffled) == release_id(release))

edited = dict(release, prompt="menu_answer_v15")
print(release_id(edited))

Run it and you should see these three lines, on any machine:

481aced016
True
8e3ea1d8fb

The id keeps the first 10 hexadecimal characters (digits 0 to 9 and letters a to f) of the fingerprint. That gives about 1.1 trillion possible ids, far more than any team will ship. One trap remains: the text "0.2" in quotes and the number 0.2 give different ids. Write each value the same way every time.

Now the Saturday question has a one-line answer. Look up the release id on the bad reply, and compare it with the release before the three changes.

Move it through three environments: try it in development, prove it in staging against the eval set, then give it a small share of production traffic. Each step can stop the change.

Staging should match production closely: the same model version, the same settings and a fresh copy of the real index. A prompt tested against last month's menu proves little about this month's.

The eval gate is a small function

The first question box is a function like the one below. The eval set has three categories: 25 allergen questions, 40 menu-fact questions and 15 tone questions. Each score is the share answered correctly, from 0 to 1.

MUST_PASS = ["allergens"]           # any mistake blocks the release
NO_WORSE = ["menu_facts", "tone"]   # may not drop below the live release

def can_promote(new, live, slack=0.01):
    for name in MUST_PASS:
        if new[name] < 1.0:
            return False, f"{name}: {new[name]:.1%}, needs 100%"
    for name in NO_WORSE:
        if new[name] < live[name] - slack:
            return False, f"{name} fell from {live[name]:.1%} to {new[name]:.1%}"
    return True, "ok to start the canary"

live = {"allergens": 25 / 25, "menu_facts": 38 / 40, "tone": 14 / 15}
new = {"allergens": 25 / 25, "menu_facts": 37 / 40, "tone": 14 / 15}
print(can_promote(new, live))
print(can_promote(new, live, slack=0.03))
print(can_promote(dict(new, allergens=24 / 25), live))

Run it and you should see:

(False, 'menu_facts fell from 95.0% to 92.5%')
(True, 'ok to start the canary')
(False, 'allergens: 96.0%, needs 100%')

MUST_PASS holds categories where one mistake blocks the release, so allergens need 100 percent. A perfect 25 out of 25 proves only those 25 questions, not that the bot never errs on allergens. NO_WORSE lets other scores slip by the slack, here 1 percentage point by default.

Look at the first line. With 40 menu-fact questions, one question is worth 2.5 points, so a 1-point slack blocks any single extra miss. That is strict, and model answers can vary between runs.

Pick the slack on purpose. Either set it to at least one question's worth, as the second call does with 0.03, or rerun the eval a few times and compare averages.

How big does a canary need to be?

A canary needs enough answers to tell a real change from luck, and that is usually more than one busy evening. The function below estimates how many canary answers you need.

It uses the standard normal approximation for testing one rate against a known rate. It is two-sided (it flags a rise or a fall) at a 5 percent false-alarm level, with 80 percent power, the chance of catching a real change of that size.

import math
from statistics import NormalDist

def canary_size(live_rate, bad_rate, alpha=0.05, power=0.8):
    z_alpha = NormalDist().inv_cdf(1 - alpha / 2)
    z_power = NormalDist().inv_cdf(power)
    spread = (z_alpha * math.sqrt(live_rate * (1 - live_rate))
              + z_power * math.sqrt(bad_rate * (1 - bad_rate)))
    return math.ceil((spread / (bad_rate - live_rate)) ** 2)

print(canary_size(0.03, 0.06))
print(canary_size(0.03, 0.04))
print(canary_size(0.001, 0.002))

Run it and you should see:

318
2493
9911

Say the live release gets thumbs-down on 3 percent of answers (made up). It serves 95 percent of traffic, about 5,700 answers a day, so we treat its rate as known. Catching a jump to 6 percent needs 318 canary answers.

At 5 percent of 6,000 questions, the canary sees 300 answers a day, so 318 takes just over a day. In our simulation of 4,000 runs, 318 answers caught the jump 81 percent of the time.

A dinner rush of 120 canary answers caught it only 44 percent of the time. A smaller jump, 3 to 4 percent, needs 2,493 answers, about eight days.

The last line matters most. A rare mistake rising from 0.1 to 0.2 percent needs 9,911 answers, about 33 days of canary traffic. At rates this rare the normal approximation runs a little low. That is why rare, severe mistakes like wrong allergen answers belong in the eval set, not the canary. Watching three signals at once raises the false-alarm chance, so treat these sizes as a floor.

In a small company, LLMOps does not need its own team. It is a set of duties shared by a few people, and every duty needs a clear owner.

• AI or ML engineer: owns prompts, model choice, retrieval and the eval set.
• Backend or platform engineer: owns environments, deployment, logging and spending limits. In a small company, this is often the same person.
• Domain expert: writes and checks the hard test questions. For the menu assistant, the head chef's team checks every allergen answer in the eval set.
• Product owner: decides what good enough means, and which mistakes are never acceptable.
• Leadership: sets risk limits and reviews incidents, such as signing off any change to how allergens are answered.

The 2022 MLOps paper also treats roles as part of the practice, naming seven, from business stakeholder to ML engineer. LLMOps is about people as much as software.

A useful way to measure progress is four rough levels, from editing prompts live in production to running every change through tests, canaries and feedback. These levels are a rule of thumb, not an official standard.

• Level 0, ad hoc: prompts edited live, no eval set, and no record of what changed. The menu assistant was here on that Saturday.
• Level 1, tracked: prompts and settings live in files, the model version is pinned (fixed to one exact version), and every answer carries a release id.
• Level 2, tested: an eval set runs before every release, staging matches production, and a gate can block a change.
• Level 3, closed loop: canaries for every change, live monitoring, and feedback that becomes new tests every week.

The jump from level 0 to level 1 takes a few days, and it is the one that answers which change caused this. Move higher as the cost of a bad answer grows.

Risk guidance points the same way. On 26 July 2024, NIST, the US standards agency, released a Generative AI Profile, NIST AI 600-1, for its AI Risk Management Framework. The framework is intended for voluntary use.

NIST says the profile can help organisations identify risks unique to generative AI and proposes actions to manage them. For the menu assistant, the team set level 2 as the floor for anything touching allergens, and level 1 for everything else.

A minimum LLMOps setup has five parts: prompts in files, a pinned model version, a small eval set, a release id on every logged answer, and one-step rollback. A small team can build it in about a week.

1. 1

   Day 1: move prompts into files

   Store each prompt in the code repository with a version in its name. Delete any prompt text pasted inside the code.

2. 2

   Day 2: pin the model

   Replace the default model setting with the exact version name your service offers. Write settings such as temperature into the same release file.

3. 3

   Day 3: write 50 to 100 test questions

   Start from real questions in the logs. Add every past complaint, and mark the categories that must never fail, like allergens.

4. 4

   Day 4: log the release id

   Save the release id, the question, the retrieved cards and the answer for every request. Mask phone numbers and addresses before saving.

5. 5

   Day 5: add the gate and practise rollback

   Run the eval set before each release and block it on a failure. Practise one rollback in staging, so the steps are known before you need them.

To check day 4 worked, ask the app one question and open its log line. You should find a 10-character release id that matches the one your release file produces.

Leave the rest for later. Dashboards, automatic canaries and tests where a model grades the answers are worth it once the basics hold. A team with fancy tools but no release id still cannot say which change broke what.

Explain LLMOps as the loop that keeps an LLM feature safe to change: version the release, test it, roll it out slowly, watch it and learn from users.

1. 1

   Name the moving parts

   Say what can change an answer: prompt, model version, settings, retrieval index and code. Point out that only one of these is ordinary code.

2. 2

   Bundle them into a release

   Describe a release id that is logged with every answer, so any bad reply can be traced to its cause.

3. 3

   Gate with an eval set

   Mention must-pass categories, like allergens here, and a no-worse rule with a slack you chose on purpose.

4. 4

   Roll out slowly

   Staging that matches production, then a canary sized by how many answers it needs, then full traffic. Rollback is a config switch.

5. 5

   Close the loop

   Monitor quality, cost and speed, and turn failures into new test questions.

6. 6

   Size the effort

   Say what you would build first for a small team, and what you would add as traffic and risk grow.