LLM Structured Output: JSON Schemas That Don't Break

Your code breaks, because the model can say the same thing in many ways and your code understands only one. LLM structured output fixes this by making the reply a fixed shape. A parser is the code that reads the reply and pulls out the pieces it needs.

Picture a common setup. A small team builds a tool that reads GST invoices for accounting firms and turns them into clean rows for tax returns. The setup is made up, but the problem is real.

One accountant's office types details from about 1,500 invoices a month into a spreadsheet, at around 3 minutes each. That is 75 hours of typing, and one wrong digit can cause trouble in a tax return.

Here is one invoice. A software firm in Coimbatore billed a garment exporter in Tiruppur for a year of software support. Invoice KT/2026/0457, dated 5 March 2026, has a taxable value of Rs 50,000.

Both firms are in Tamil Nadu, so the tax is CGST of Rs 4,500 plus SGST of Rs 4,500, and the total is Rs 59,000. CGST and SGST are the central and state shares of GST on a sale inside one state. A sale to another state pays IGST instead.

The team's first version sends the invoice text to a model with a short prompt asking for the invoice number, date, amounts and total. Then a regular expression, a pattern-matching rule for text, picks the values out of the reply.

It worked on Monday. On Tuesday the model wrote "Inv No" instead of "Invoice number", and the pattern found nothing. On Wednesday it wrote the total as "Rs 59,000 only", and the pattern stopped at the comma and saved 59 rupees.

Nothing was wrong with the model. It was being helpful, in slightly different words each time. The code expected one exact shape, and nothing made the model stick to it.

LLM structured output is a model reply in a fixed shape that your code can read without guessing. Usually that shape is JSON that matches a schema you wrote.

JSON is a plain-text format for data, made of names and values inside curly brackets. Python turns JSON text into a dictionary with one call to json.loads. A schema is a written description of the shape your data must have: the field names, their types and their limits.

This is the reply the team wants for the Coimbatore invoice. Every amount is a plain number, and the unreadable list is empty because the model could read everything.

{"schema_version": 2, "invoice_number": "KT/2026/0457",
 "invoice_date": "2026-03-05", "seller_gstin": "33ABCDE1234F1Z5",
 "buyer_gstin": "33PQRST6789K1Z2", "supply_type": "intra_state",
 "taxable_value": 50000, "cgst": 4500, "sgst": 4500, "igst": 0,
 "total": 59000, "unreadable": []}

So the team changed the prompt to say: reply with JSON only, using exactly these field names. Things got much better, but not perfect. About 1 in 40 replies (a made-up rate) still broke the code that week.

These are the kinds of failures they saw. We ran a copy of each one through the checker shown later, and each is caught:

• Extra words around the JSON, like a friendly sentence before it.
• Numbers written as text, such as "59,000" with a comma, which your code cannot add up.
• A renamed field, like invoice_no instead of invoice_number.
• A missing field, because the invoice had no buyer GSTIN, the 15-character GST registration number.
• Broken JSON, such as a missing closing bracket, when a long reply was cut off at the length limit.

At 1,500 invoices a month, 1 in 40 means about 38 broken records, and someone has to find and fix each one. Asking nicely in the prompt is not enough.

Good systems add two firmer tools. The model service can enforce your schema while the model writes, and your own code validates the reply before trusting it. Both need a schema, so that comes first.

JSON Schema is an open specification for writing schemas, and each schema is itself written in JSON. It describes your output as rules: which fields exist, their types, which are required and what values are allowed.

The current published version is called 2020-12, as of 19 October 2026. Most languages have a library that checks data against it. A keyword is one rule name inside the schema, like type or required.

The team wrote one schema for every invoice, with twelve fields. The table shows the keywords it uses.

Here is the whole schema. It is the exact file our tests load.

{
  "type": "object",
  "properties": {
    "schema_version": {"const": 2},
    "invoice_number": {"type": ["string", "null"]},
    "invoice_date": {"type": ["string", "null"], "format": "date"},
    "seller_gstin": {"type": ["string", "null"], "pattern": "^[0-9]{2}[A-Z0-9]{13}$"},
    "buyer_gstin": {"type": ["string", "null"], "pattern": "^[0-9]{2}[A-Z0-9]{13}$"},
    "supply_type": {"enum": ["intra_state", "inter_state", null]},
    "taxable_value": {"type": ["number", "null"], "minimum": 0},
    "cgst": {"type": ["number", "null"], "minimum": 0},
    "sgst": {"type": ["number", "null"], "minimum": 0},
    "igst": {"type": ["number", "null"], "minimum": 0},
    "total": {"type": ["number", "null"], "minimum": 0},
    "unreadable": {"type": "array", "items": {"type": "string"}}
  },
  "required": ["schema_version", "invoice_number", "invoice_date",
               "seller_gstin", "buyer_gstin", "supply_type", "taxable_value",
               "cgst", "sgst", "igst", "total", "unreadable"],
  "additionalProperties": false
}

Two rules do most of the safety work. The required rule makes every field appear, so nothing goes missing quietly. Setting additionalProperties to false blocks unknown fields, so a renamed field like invoice_no fails loudly instead of hiding.

A type can also be a list of types. The team sets buyer_gstin to string (a piece of text) or null (JSON's word for no value). Shops that sell to ordinary customers often have no buyer GSTIN, so null is a correct answer there.

Constrained decoding forces the shape by blocking, at every step, any token that would break the schema. A token is a small piece of text, roughly a word or part of a word. The model can only pick tokens that keep the reply on track.

A model writes its reply one token at a time. At each step it scores every token it knows, and one is picked. This picking step is called decoding.

Constrained decoding adds a check before that pick. The engine turns your schema into a grammar, a set of rules for valid text, and then into a map of states. Each state says which tokens may come next.

Say the reply so far ends with the total field name and a colon. The schema says total is a number or null, so only tokens that start a number, or the word null, are allowed.

Every other token gets its chance set to zero. Engineers call this masking. A 2023 paper on guided generation showed how to make it fast, by working out in advance which tokens each state allows.

Think of a school exam. Asking for JSON in the prompt is like telling students to write only A, B, C or D on a blank page. Most will, but not all.

Constrained decoding hands them an OMR sheet instead: a bubble sheet that a machine reads in seconds. The prompt maps to the instruction, the schema maps to the printed sheet, and masking maps to the bubbles. The only marks students can make are the bubbles.

Where the picture breaks: an OMR sheet is printed in full before the exam. Constrained decoding works out the allowed next tokens one step at a time, and some engines leave a few of your rules off the sheet.

This runs inside the model service, or inside your inference engine if you host the model yourself. An inference engine is the software that loads and runs a model, such as vLLM. Services name the feature differently, often structured outputs or a JSON schema mode.

Some services also offer a JSON mode. Read what it promises: a guarantee of parseable JSON is not a guarantee of your field names or types.

Constrained decoding is a strong tool, but it has limits you should know:

• It controls shape, not truth. The model can still misread Rs 59,000 as Rs 5,900 and write a perfectly valid number.
• Engines do not enforce every keyword. A 2025 benchmark study tested six structured output systems on about 10,000 real-world schemas. None supported every feature in the official JSON Schema test suite.
• A reply can still be cut off. If it hits the output length limit, the JSON stops halfway and will not parse.

You validate output in three steps: parse it, check it against the schema, then check your business rules. Validation means checking data against rules before you trust it. Business rules say what the data means, like totals adding up.

Do this even when the service enforces your schema, because the right shape is not the right answer. Go back to the OMR sheet: the machine checks that one bubble is filled, not that it is the right bubble. Your validation code is the teacher who checks the answers.

OWASP, a non-profit group that publishes security guidance, lists improper output handling among its top 10 risks for LLM applications (2025 list). That means passing model output to other systems without enough checks.

Its first prevention step is to treat the model like any other user, with a zero-trust approach: nothing it writes is trusted until it is checked. The team's replies end up in tax returns, so the team checks every one.

Here is the checker. It uses jsonschema, an open-source Python library that checks data against a JSON Schema. We ran it on Python 3.13 with jsonschema 4.26.0. It returns the data and a list of errors, and an empty list means all is well.

import json
import math
from jsonschema import Draft202012Validator

with open("invoice_schema.json") as f:
    INVOICE_SCHEMA = json.load(f)
validator = Draft202012Validator(            # built once
    INVOICE_SCHEMA, format_checker=Draft202012Validator.FORMAT_CHECKER)
AMOUNTS = ("taxable_value", "cgst", "sgst", "igst", "total")

def reject_constant(name):         # Python's json accepts NaN by default
    raise ValueError(f"{name} is not allowed in JSON")

def finite_float(text):            # 1e400 would become infinity
    value = float(text)
    if not math.isfinite(value):
        raise ValueError(f"{text} is too large")
    return value

def check_invoice(reply_text):
    try:
        data = json.loads(reply_text, parse_constant=reject_constant,
                          parse_float=finite_float)
    except ValueError as err:
        return None, [f"Not valid JSON: {err}"]
    errors = [f"{e.json_path}: {e.message}" for e in validator.iter_errors(data)]
    if errors:
        return None, errors
    errors += [f"$.{k}: null but not listed in unreadable"
               for k in AMOUNTS if data[k] is None and k not in data["unreadable"]]
    amounts = [data[k] for k in AMOUNTS]
    if None not in amounts:                          # every amount was read
        taxable, cgst, sgst, igst, total = amounts
        if abs(taxable + cgst + sgst + igst - total) > 1:
            errors.append("$.total: does not equal taxable value plus tax")
    return data, errors

Three parts matter most. The call to iter_errors finds every schema problem at once, not just the first. Each message starts with a JSON path, like $.total, which points to the place in the reply that failed.

Passing FORMAT_CHECKER switches on the date check that the spec leaves off. The reject_constant and finite_float helpers refuse NaN, Infinity and numbers so large they turn into infinity. The engineer sections explain both traps.

The last check is a business rule a schema cannot express: taxable value plus all three taxes must equal the total.

The totals rule runs only when no amount is null. The prompt asks for 0 when a tax does not apply, so null always means unreadable. The gap of 1 rupee allows for rounding.

Check that it works on your machine

Save the schema as invoice_schema.json, the reply above as reply.json and the checker as check_invoice.py. Then run this small script, which breaks the reply in four ways.

from check_invoice import check_invoice

reply = open("reply.json").read()
print(check_invoice(reply)[1])
print(check_invoice("Sure! " + reply)[1])
print(check_invoice(reply.replace("59000", "5900"))[1])
print(check_invoice(reply.replace("2026-03-05", "2026-02-30"))[1])
print(check_invoice(reply.replace("59000", "NaN"))[1])

You should see five lines. The first is an empty list, for the good reply. Then a JSON parse error at line 1 column 1, the totals error, a date error for 2026-02-30 and a NaN error. The whole run takes well under a second.

The retry-with-error loop sends the exact validation errors back to the model and asks it to fix its reply, a small, fixed number of times. If the reply still fails, a person takes over.

It works because the errors are specific. A vague message like "wrong reply" gives the model little to go on. A message that names $.total and says it does not add up tells the model which field to read again.

Here is what happened to the Coimbatore invoice when the scan was blurry.

1. 1

   Try 1

   The model writes total as 5900. The schema check passes, because 5900 is a number. The totals check fails: 50,000 plus 9,000 in tax is 59,000, not 5,900.

2. 2

   Feedback

   The code adds the failed reply and the error to the prompt, and asks for corrected JSON only.

3. 3

   Try 2

   The model reads the total line again and writes 59000. Every check passes.

4. 4

   Save

   The invoice goes into the table. The log keeps both tries.

Here is the loop in Python. Two helpers stand in for your own code. The helper ask_model sends the prompt and invoice text to the model service and returns the reply. The helper send_to_review puts an invoice in the staff review queue.

from check_invoice import check_invoice

def extract_invoice(invoice):
    prompt = EXTRACT_PROMPT
    for attempt in range(3):                  # 1 first try + 2 retries
        reply = ask_model(prompt, invoice.text)
        data, errors = check_invoice(reply)
        if not errors:
            if data["unreadable"]:            # valid, but some fields unclear
                send_to_review(invoice.id, "Check: " + ", ".join(data["unreadable"]))
            return data
        prompt = (EXTRACT_PROMPT
                  + "\nYour last reply was:\n" + reply
                  + "\nIt had these problems:\n" + "\n".join(errors)
                  + "\nReturn the corrected JSON only.")
    send_to_review(invoice.id, "Failed checks 3 times")
    return None

What just happened? We ran this loop with a scripted fake model that first wrote 5900, then 59000. It made two model calls and saved the invoice with a total of 59000.

A second script fed it a reply that never parses. It made three calls, returned None and sent the invoice to review with the note "Failed checks 3 times".

A reply that passes every check still goes to review if its unreadable list is not empty. The rest of the data is saved, so a person only fills the gaps. On a failure, the new prompt holds the failed reply as well as the errors, so the model can fix it instead of starting over.

It should say so plainly, with null or a list of unreadable fields, instead of guessing. Your schema must make that honest answer possible, or the model will invent something that fits.

This is where constrained decoding can work against you. If a field allows only a number and the scan is smudged, the engine still forces some number out. A sentence like "I can't read this" is not an allowed token path.

In a small test, a scripted stand-in for a model preferred the words I can't read. The engine blocked them, and the reply came out as {"total": 90}, a valid but invented number.

So give the model a valid way to say it does not know.

The team's schema does this in four ways:

• Fields that may be empty or unreadable, like buyer_gstin, invoice_date and each amount, allow null as well as their normal type.
• Every field is still required, so the model must write null out loud. A field that is simply missing could mean anything.
• A field called unreadable holds a list of the field names the model could not read clearly, such as invoice_date.
• The prompt says: if you cannot read a value, write null and add its name to unreadable. Never guess an amount. The checker flags an amount that is null but not listed.

Some teams also ask the model for a confidence score per field. Treat it as a hint at most: it is generated text, not a measured probability. Checks you compute yourself, like the totals rule, deserve far more trust.

A null value is not a dead end. It sends the invoice to the review queue with the exact fields to check. If 5 percent of invoices land there, staff check about 75 invoices a month instead of typing all 1,500.

Beginners can stop here: you now know the contract, the enforcement, the checks and the retry loop. The next sections are for engineers. They cover what our tests showed that the docs barely mention, Pydantic models, versioning, testing and the interview answer.

It depends on the engine, and some rules are silently left out. We checked one of the four open-source grammar libraries that vLLM 0.32.0 lists as dependencies, at the exact version vLLM pins.

vLLM 0.32.0 was the newest release on 19 October 2026. We called the library directly, not through vLLM, so treat this as a check of the library.

We built the engine's pattern from the invoice schema and asked whether it accepts replies that break each rule. Pause and predict: the schema says cgst has a minimum of 0. Can the engine write minus 4500?

So yes, it can. The engine kept the shape rules and dropped the number limit and the calendar check. Your own validator catches both, which is one more reason it must run on every reply.

The 2025 benchmark study calls this under-constraining: an engine allows output that the schema forbids. It found that this varied between the six systems it tested.

vLLM's default backend setting is auto, which picks a grammar library per request (its docs, as of 19 October 2026). In vLLM 0.32.0's source code, auto tries a different library first. It falls back to the one we tested only in narrow cases that depend on the schema and the model. So check which backend your server actually uses, pin it and rerun your test set when you upgrade.

Two traps passed bad values in our runs on Python 3.13 and jsonschema 4.26.0. Each one looks safe until you test it.

NaN slips through Python's json module

By default, json.loads accepts NaN and Infinity, which the JSON standard does not allow, as Python's own docs say. NaN means not a number. A reply with total set to NaN parsed, passed the schema check and slipped past the totals rule.

That is why the checker passes parse_constant to json.loads. It rejects those words with the message "NaN is not allowed in JSON".

A number too large for a float, like 1e400, also becomes infinity, with no NaN word to catch. Without finite_float, a reply with 1e400 as both taxable value and total passed every check in our run. Infinity minus infinity is NaN, and a NaN gap is never greater than 1, so the totals rule stayed quiet. The finite_float helper rejects any number that is not finite.

A date-time check that quietly passes

The jsonschema docs say some formats need extra packages, and that validation succeeds without error if one is missing. The date format needs nothing extra. The date-time format needs rfc3339-validator.

Without that package installed, we checked the string "yesterday" against date-time with the format checker switched on. It passed with no error. Install the package, or test each format you rely on.

You describe the fields as a Python class with type hints, then export the JSON Schema from it. Pydantic is a popular open-source Python validation library. The class becomes one source of truth for the schema you send and the checks you run.

This short version of the invoice ran on Pydantic 2.14.1. It prints the required list, the rule for total and the additionalProperties setting.

from typing import Literal, Optional
from pydantic import BaseModel, ConfigDict

class Invoice(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True, allow_inf_nan=False)
    invoice_number: Optional[str]
    total: Optional[float]
    supply_type: Optional[Literal["intra_state", "inter_state"]]
    unreadable: list[str]

schema = Invoice.model_json_schema()      # send this to the model service
print(schema["required"])
print(schema["properties"]["total"])
print(schema["additionalProperties"])

Expect three lines: all four field names in the required list, total as a number or null, and False. Three settings in that class matter.

• extra="forbid" exports additionalProperties as false. The Pydantic default ignores extra fields silently.
• A field typed as optional but given no default stays required. Give total a default of None and it drops out of the required list, so the model may skip it.
• allow_inf_nan=False rejects NaN and Infinity. The documented default is True, and our run with the default accepted NaN from JSON text.

strict=True keeps the class and the exported schema in agreement. Without it, Pydantic turned the string "59000" into the number 59000.0. The exported JSON Schema rejected that same string.

You version a schema by giving it a number that every saved reply carries. You test it by running a fixed set of known examples through the whole pipeline before every change.

How do you change a schema without breaking old data?

Add new fields that may be null, and never rename, retype or remove a field in place. Adding a new subject row to a report card is harmless, because old cards leave it blank. Renaming Maths to Numeracy breaks every comparison with old cards.

Three months in, the accountant asks for the place of supply, the state where a sale is taxed. The team adds a place_of_supply field that may be null, and the schema becomes version 3.

• Save the schema version, the prompt version and the model version with every invoice record.
• Keep every old schema in a schema store, so old records can still be read and checked.
• Change the schema, the prompt and the checker together, in one release, so they never disagree.
• For a breaking change, write a migration, a small script that converts old records to the new shape.

How do you test structured output?

You test it with a fixed set of documents whose correct answers you already know. The team keeps 60 real invoices that staff already checked by hand. Every change to the schema, the prompt or the model runs against all 60 before release.

• First-try pass rate: how many replies pass every check without a retry.
• Field accuracy: how often each field matches the hand-checked answer, per field. A wrong GSTIN and a wrong total cost different amounts.
• Review rate: how many invoices end up with a person.
• Retries and cost per invoice.

The team also keeps unit tests, small automatic tests for one function, for check_invoice. They use broken replies saved from real failures, like text before the JSON. They need no model, so they run in seconds.

Explain it as a contract plus checks: a schema the model must follow, code that validates every reply, a small retry limit and a person for whatever is left. Then show how you would version and test it.

1. 1

   Name the reader of the output

   Say which program reads the output and what breaks if it is wrong. Here it feeds a tax return, so wrong numbers are costly.

2. 2

   Write the schema

   List the fields, types and allowed values. Make every field required, and allow null wherever the answer may be unknown.

3. 3

   Enforce the shape

   Use schema-enforced output when the model service supports it. Otherwise ask for JSON in the prompt and expect more retries.

4. 4

   Validate in code

   Parse the reply, check the schema, then check business rules like totals. Treat model output as untrusted input.

5. 5

   Retry, then hand over

   Send exact errors back for one or two retries, then send the record to a person.

6. 6

   Version and test

   Store a version with every record and run a hand-checked test set on every change. Watch pass rate and review rate in production.