AI Decision Models Explained: Typed Choices Instead of Generated Text

You get text back when your code wanted a choice. A chat model writes its reply one small piece at a time, so it must spell out its pick, and your code must parse and check it. AI decision models skip the writing and return one of your options with a probability for each.

Picture a common setup. A grocery delivery app's support inbox gets about 30,000 customer messages a day. Each message must reach one of five teams: payments, delivery, app_bug, account or other. The setup and its numbers are made up, but the problem is real.

Today a large hosted chat model reads each message and writes a team name. Each call takes about 1.2 seconds, and the bill is about Rs 2,700 a day. Roughly one reply in 300 is a label that is not on the list, such as "payment_issue_maybe", and the code has to catch it.

The bigger gap is trust. The reply says "payments" but not how sure the model is. The team cannot tell a clear UPI payment complaint from a message that mixes cash, refunds and a late rider.

Six words carry the rest of this guide. Each one is explained in plain terms here.

• Options: the fixed list of answers you allow, such as the five team names. The model may only pick from this list.
• Probability: a number from 0 to 1 that says how likely an option is. Across all options, the probabilities add up to 1.
• Token: a small piece of text, such as a word or part of a word. Chat models write their replies one token at a time.
• Forward pass: one trip of the input through the model's layers, from text in to numbers out. A chat model runs roughly one more pass for every token it writes.
• Threshold: a cut-off you choose. Answers whose top probability falls below it go to a person instead of being acted on automatically.
• Calibration: whether the probabilities match reality. If a calibrated model says 0.8 on 100 messages, it should be right on about 80 of them.

AI decision models answer a typed question, one whose answer must take a fixed form, by choosing from options you supply. For a choice question, the answer is always one of those options, with a probability for each. They write no replies, code or explanations.

Think of a railway ticket checker with three rubber stamps: valid, upgrade and fine. The checker reads your ticket and stamps one box, but never writes you a letter. Your options are the stamps, and the ticket is the text you send.

Where the picture breaks: a checker stamps one box and moves on. A decision model also reports how strongly it leaned towards every stamp. That lean is what lets your code decide when to ask a person.

Some vendors and runtimes call them System One models. The name comes from the fast, intuitive thinking that Daniel Kahneman described in his book Thinking, Fast and Slow. The diagram shows where one sits in the support flow.

It reads the text and the options, gives each allowed option a score and turns the scores into probabilities. The top option becomes the answer. Nothing outside your list can win.

One read, one score per option

A chat model writes token by token, and each new token needs another forward pass. A decision model skips that writing loop. The llama.cpp team says its server returns a probability for each option in a single forward pass.

The scores only cover the options you sent, which is why "payment_issue_maybe" can never appear. A final step, called normalising, rescales the scores so they add up to 1. The UPI message might come back as payments 0.93 and delivery 0.04 (made up).

Why the option descriptions matter

The model judges each option partly by the words you attach to it. In the llama.cpp team's own example, a small model sent a billing complaint to shipping when the options were bare labels. With a short description on each option, it picked billing at 0.99.

So write each description the way you would brief a new support agent. Say what belongs in that queue, in words customers actually use.

Two research designs from September and October 2026

A paper posted on 20 September 2026 describes a decision model with 2 billion parameters, the learned numbers inside a model. It reads its answer from a hidden state, the model's internal list of numbers at one chosen position. It considers only the options the caller declared and generates zero output tokens.

A second paper, posted on 13 October 2026, encodes the request and each option separately with one shared encoder, the part of a model that turns text into numbers. It combines them in a way that treats every position alike, so option order should not matter, and the authors report no flipped decisions in their reordering tests. A separate coverage estimate lets it answer that none of the options fits.

Ollama's API has three question types: choice (pick one option), noul and score (a level on an ordered scale). Noul is the API's name for a yes-or-no question, answered as the probability that a statement is true. One request can hold 1 to 64 named questions about the same text.

Questions do not see each other's answers. In a choice answer, Ollama's docs say ties follow the model's option order. Whether that means the order in your request is undocumented as of 19 October 2026, so pin your version, meaning stay on the version you tested. A score answer is a fraction from 0 up to the number of levels minus 1, not a rounded level.

Use a decision model to route among fixed options, a chat model when you need written text and a trained classifier for stable labels with long history. A classifier is a small model trained only to sort text into fixed labels.

A decision model cuts the setup cost, not the testing cost. You can rename a team in the next request, but you still need labelled messages to know whether the answers are good.

Install Ollama v0.35.0 or later, pull a decision model and send a POST request, an HTTP call that carries data, to /v1/systemone. Running locally keeps customer messages on your own machines, which is why the team tries it first.

1. 1

   Upgrade the runtime

   Install or upgrade Ollama to v0.35.0 or later. That release added the /v1/systemone endpoint.

2. 2

   Pull a decision model

   Download one with ollama pull followed by the model's name, as the v0.35.0 release notes show. Check its page for how many options it accepts.

3. 3

   Write the question

   Put the customer message in state, the field for the text being judged. Add one choice question with clear instructions and a short description for each team.

4. 4

   Read the typed answer

   Take choice, probabilities and confidence from the JSON reply, a common data format. There is no free text to repair.

5. 5

   Set a threshold from data

   Run labelled past tickets through it, say 2,000 of them (made up). Pick the cut-off that keeps wrong routes rare, and send the rest to people.

Here is the team's first request in Python. The model name is a placeholder for whichever decision model you pulled.

import requests

ENDPOINT = "http://localhost:11434/v1/systemone"
TEAMS = {
    "payments": "UPI, wallet, card payments and refunds",
    "delivery": "Late, missing or damaged orders",
    "app_bug": "App crashes, errors and screens that do not load",
    "account": "Login, OTP and profile changes",
    "other": "Anything that fits none of the above",
}

body = {
    "model": "your-decision-model",
    "state": "Paid Rs 640 by UPI but the order still shows unpaid.",
    "questions": {
        "team": {
            "type": "choice",
            "instructions": "Which support team should handle this message?",
            "criteria": TEAMS,
        },
    },
}
# The first call may also wait for the model to load, so allow time.
reply = requests.post(ENDPOINT, json=body, timeout=60)
reply.raise_for_status()
answer = reply.json()["answers"]["team"]
print(answer["choice"], round(answer["confidence"], 2))
print(answer["probabilities"])

The criteria field holds the allowed options, and the answer comes back under the question's key, team. On the team's test machine it arrives in about 40 milliseconds (made up), against 1.2 seconds before. Much of that gap comes from a small local model replacing a large hosted one. In the llama.cpp team's timings on one GPU, a graphics chip that speeds up models, decision models took 3 milliseconds at 144 million parameters and 43 at 27 billion.

Since v0.40.4 (12 October 2026), you can also ask a decision model a quick yes-or-no question from the terminal. The release notes show ollama run with the model name, an --ask flag and a ticket file piped in.

It tells you how concentrated the probabilities are, not how likely the answer is to be right. Entropy measures how spread out the probabilities are. Ollama's docs define confidence as 1 minus the entropy divided by ln(N), where N is the number of options. ln(N), the natural logarithm of N, is the largest entropy N options can have.

If all five teams get 0.2 each, confidence is 0. If one team gets almost everything, confidence nears 1. The team rebuilds the formula to see this.

import math

def confidence(probs):
    entropy = -sum(p * math.log(p) for p in probs.values() if p > 0)
    return 1 - entropy / math.log(len(probs))

def route(probs, threshold=0.8):
    top = max(probs, key=probs.get)
    return top if probs[top] >= threshold else "human_triage"

clear = {"payments": 0.93, "delivery": 0.04, "app_bug": 0.01, "account": 0.01, "other": 0.01}
mixed = {"payments": 0.46, "delivery": 0.41, "app_bug": 0.07, "account": 0.03, "other": 0.03}

for probs in (clear, mixed):
    print(round(confidence(probs), 2), route(probs))

Run it and you should see 0.79 payments, then 0.3 human_triage. The clear message routes itself, and the mixed one goes to a person. The route function uses the top probability, and confidence is logged as a separate signal.

To check the formula yourself, feed in the docs' example probabilities of 0.0125, 0.9781 and 0.0093. You should get about 0.891. The docs show 0.8906 because their inputs are rounded to four places.

Pick it from labelled data, not from the model's numbers alone. Run past messages whose correct team you know, record the top probability and whether it was right, then compare cut-offs.

# (top probability, was the top choice right?) for labelled past tickets
results = [(0.97, True), (0.95, True), (0.93, True), (0.91, False), (0.88, True),
           (0.84, True), (0.81, True), (0.76, False), (0.72, True), (0.64, False),
           (0.58, True), (0.47, False)]

for threshold in (0.6, 0.8, 0.9):
    auto = [ok for p, ok in results if p >= threshold]
    print(f"{threshold}: auto-routed {len(auto)} of {len(results)}, wrong {auto.count(False)}")

band = [(p, ok) for p, ok in results if p >= 0.9]
said = sum(p for p, _ in band) / len(band)
right = sum(ok for _, ok in band) / len(band)
print(f"0.9 and above: said {said:.2f} on average, right {right:.2f} of the time")

What just happened? At 0.6, 10 of 12 tickets route themselves and 3 are wrong. At 0.8, 7 route and 1 is wrong. At 0.9, only 4 route and 1 is still wrong, so raising the bar trades automation for fewer mistakes.

The last line is a calibration check. At 0.9 and above, the model said 0.94 on average but was right only 0.75 of the time. Twelve made-up tickets prove nothing, so run this on hundreds of real ones per team.

Repeat the check whenever you switch models. In the llama.cpp team's post, the same vague ticket scored 0.25 on one model and 0.80 on another, so a cut-off does not carry over.

It showed that describing the same two tracks in the other order changed the hosted service's choice in about one pair in seven. It is one blogger's experiment, published on 14 October 2026, not a general law.

The post put 5,240 trolley-problem setups to each of 14 decision models, and a tech news site covered it two days later. The hosted service's model changed its choice in 370 of 2,620 pairs. All but one of the other models also changed some answers this way. The author warns against sweeping conclusions about AI safety.

For a support inbox, the order of facts in a message or in your instructions may sway a choice. Customers tell the same story in many orders, so add reworded copies of the same problem to your test set.

Shuffle the option list and check that the top choice holds. The trolley post did not test this, but two facts make it worth checking. Ollama's docs say ties follow the model's option order, and the 13 October paper builds order independence into its design.

The test below sends one message with the options in three random orders. It reports whether the top choice changed between them.

import random
import requests

def top_choice(state, criteria):
    question = {"type": "choice", "criteria": criteria,
                "instructions": "Which support team should handle this message?"}
    body = {"model": "your-decision-model", "state": state, "questions": {"team": question}}
    reply = requests.post("http://localhost:11434/v1/systemone", json=body, timeout=60)
    reply.raise_for_status()
    return reply.json()["answers"]["team"]["choice"]

def flips(state, criteria, orders=3, seed=7):
    rng = random.Random(seed)
    keys = list(criteria)
    seen = set()
    for _ in range(orders):
        rng.shuffle(keys)
        seen.add(top_choice(state, {k: criteria[k] for k in keys}))
    return len(seen) > 1

Run flips over a few hundred labelled messages and count the True results. Messages with near-even probabilities are the likeliest to flip. The docs do not say whether the tie order is your request order, so exact ties may flip too. If flips cluster on certain messages, sharpen those option descriptions or send such messages to people.

Five details from Ollama's docs and release notes matter in production. All of them are as of 19 October 2026, so pin your Ollama version and recheck after upgrades.

• Output tokens are not always zero. Ollama counts tokens generated internally for scoring, including retries, and the docs' own example reports 1 output token.
• Shared text is counted again. When questions are scored separately, the state is counted again in input_tokens, so many questions on a long message add up.
• Input is never cut short. On a local server, text and questions over 64 KiB (65,536 bytes) return error 413. Input longer than the context window, the most the model can read at once, returns 400.
• Option limits differ by source. The docs allow 2 to 255 choice options, depending on the model, and 2 to 10 score levels. The Python client 0.6.3 README still says 2 to 26 for both.
• Cloud is now possible, with a catch. Since v0.40.5, Ollama can run decision models in its cloud, which needs a paid plan or usage credits and sends the text off your machines.

Skip one when the answer must be written, explained or calculated. A decision model only picks from the list you give it.

• Replies to customers, summaries and emails need generated text, so a chat model is the right tool.
• Multi-step arithmetic is weak. The 20 September paper reports 0.560 for its own model there, against 0.98 to 1.00 for the models it was compared with.
• Open-ended inputs can fit none of your options. Ollama's choice answer always names one of them, so keep an explicit other option and route it to people.
• Life-or-death or moral calls need people, as the trolley post's author also argues.

Frame it as a smart if-statement: a model judges the condition, but your code owns the threshold. That split, probabilities from the model and decisions from your code, is the point to land.

1. 1

   State the job

   Say the model reads text plus a fixed list of options and returns one option with a probability for each. It writes no text.

2. 2

   Draw the flow

   Show the app building state and options, the decision model scoring them and app code applying a threshold.

3. 3

   Explain why it is fast

   There is no writing loop, so it skips the extra passes a chat model needs to write the label. Add that a small model drives much of the speed, and some runtimes still spend internal scoring steps.

4. 4

   Own the threshold

   Explain that you pick the cut-off from labelled data and send low-probability cases to people or a bigger model.

5. 5

   Name the risks

   Mention sensitivity to the order of options or facts, vague option descriptions and drift, meaning accuracy slowly drops as messages change. Add shuffle tests and track accuracy by probability band.