Temperature, Top-p and Top-k: LLM Sampling Explained

Sometimes nothing goes wrong: a fresh answer is exactly what you want for a tagline. But when a bot copies an order number, a different answer is a bug. LLM temperature, top-p and top-k decide how much the answers vary.

The same prompt gives different answers because the model does not simply return its single best guess. By default it picks each token with some controlled randomness. One setting cannot suit a job that needs the same answer every time and a job that needs a new answer every time.

Picture a common setup. A small team in Mumbai writes taglines for local shops and also runs a support bot for its own customers. Both jobs call the same model with one shared settings file. The setup is made up, but the problem is real.

The support bot reads about 1,200 messages a day. It pulls the order number and the problem out of each message and saves them as JSON, a simple text format for data. Pulling fixed facts out of text like this is called extraction.

To get fresher taglines, the team raised temperature for every call. The copywriters were happy. Then about 3 in every 100 saved order records came out broken, with extra words like "probably" next to the order number.

So the team set temperature to 0 everywhere, and the records were clean again. Then a copywriter asked for taglines for a chai stall and pressed the button for more ideas. She got almost the same five lines every time.

This guide shows what each sampling setting really does, so you can give each job its own settings on purpose instead of by guessing.

You need five words: token, score, probability, decoding and sampling. Each one below comes with an example from the chai stall's taglines.

Token. A token is a small piece of text, roughly a word or part of a word. The model writes its reply one token at a time. A common word like chai may be one token, while a rare brand name may be split into several pieces.

Score. Before writing each token, the model gives every token in its vocabulary, the fixed list of tokens it knows, a score. Engineers call these raw scores logits. A higher score means the model thinks that token fits better next.

Probability. A probability is a chance, written here as a percentage. The scores get turned into percentages that add up to 100. Example: chai 48%, energy 24%, joy 16%, and so on.

Decoding. Decoding is the rule that turns those percentages into one chosen token. Temperature, top-k and top-p are all decoding settings. They change how the model picks, not what it knows.

Sampling. Sampling means picking at random, but weighted by the percentages. Picture a raffle at a housing society Diwali mela where chai holds 48 of the 100 tickets. Chai usually wins, but not always.

This guide uses that raffle as its one picture. Here is how its parts map to the real thing:

• Tickets are the percentages: more tickets means better odds of winning.
• Temperature moves tickets between holders before the draw.
• Top-k and top-p send some holders home, so their tickets leave the drum.
• The seed fixes the spin of the drum in advance, so the same tickets give the same winner.

They sit between the model's scores and the final pick. The model scores every possible token, these settings reshape and trim the odds, and one token is chosen. This repeats for every token in the reply.

Take the chai stall's tagline. Say the reply so far is "Your daily dose of" and the model must choose the next word. Say the top five scores are chai 3.0, energy 2.3, joy 1.9, masala 1.2 and sunshine 0.4.

A real model scores every token in its vocabulary, a far longer list, but five is enough to learn from. The diagram below shows one round of that choice. Watch the last arrow: the chosen token goes back in as text for the next round.

What is greedy decoding?

Greedy decoding means always taking the token with the highest score, with no randomness at all. For the chai tagline, greedy picks chai every time the scores come out the same. So the same prompt gives the same tagline, again and again.

Greedy works well when there is one right answer, like copying an order number. It works badly for creative work. The 2019 paper that introduced top-p sampling found that decoding for the most likely text gives text that is "bland and strangely repetitive".

That is exactly what the copywriter saw when she got five nearly identical taglines. The fix is not to give up on randomness, but to control it.

Temperature changes how spread out the percentages are before the pick. A low temperature makes the top token even more likely, and a high temperature gives the other tokens a bigger share.

Here is how. The model divides every score by the temperature, then turns the scores into percentages as usual. Dividing by a small number like 0.5 stretches the gaps between scores. Dividing by a bigger number like 1.5 squeezes the gaps together.

In raffle terms, temperature moves tickets around. A low temperature hands more tickets to the favourite. A high temperature shares them out more evenly. Here are the chai stall's percentages at three temperatures, rounded, from the scores above:

• Temperature 0.5: chai 72%, energy 18%, joy 8%, masala 2%, sunshine under 1%.
• Temperature 1.0: chai 48%, energy 24%, joy 16%, masala 8%, sunshine 4%.
• Temperature 1.5: chai 39%, energy 24%, joy 19%, masala 12%, sunshine 7%.

Chai stays on top at every temperature. Dividing every score by the same positive number never changes their order. It only changes how big the gaps are, so temperature changes how often the favourite wins, never which token is the favourite.

You cannot divide by zero, so temperature 0 is a special case. vLLM 0.32.0 and llama.cpp both treat it as greedy decoding, as their source code shows as of 19 October 2026. At the other end, a high temperature gives more tickets to rare tokens.

The 2019 nucleus sampling paper calls that low-chance region the unreliable tail and blames it for incoherent text. It also reports that very high temperatures hurt coherence.

Here is the temperature maths in plain Python, using only the standard math module. It takes the chai stall's scores and prints the percentages for three temperatures.

import math

def to_probs(scores, temperature=1.0):
    scaled = {w: s / temperature for w, s in scores.items()}
    biggest = max(scaled.values())
    exps = {w: math.exp(s - biggest) for w, s in scaled.items()}
    total = sum(exps.values())
    return {w: e / total for w, e in exps.items()}

scores = {"chai": 3.0, "energy": 2.3, "joy": 1.9, "masala": 1.2, "sunshine": 0.4}
for t in (0.5, 1.0, 1.5):
    probs = to_probs(scores, t)
    print(t, {w: round(100 * p) for w, p in probs.items()})

Run it with python3, and you should see these three lines (sunshine at temperature 0.5 is 0.4%, which rounds to 0):

0.5 {'chai': 72, 'energy': 18, 'joy': 8, 'masala': 2, 'sunshine': 0}
1.0 {'chai': 48, 'energy': 24, 'joy': 16, 'masala': 8, 'sunshine': 4}
1.5 {'chai': 39, 'energy': 24, 'joy': 19, 'masala': 12, 'sunshine': 7}

The line that matters most is the first one inside to_probs: it divides every score by the temperature. The rest is softmax, the standard step that turns scores into shares that add up to 1, which is 100%.

Subtracting the biggest score before math.exp avoids overflow, a number too big for the computer to store. It does not change the result.

Top-k keeps a fixed number of the most likely tokens, while top-p keeps the smallest group of top tokens whose chances add up to a chosen share, like 90%. Both cut the list before the random pick.

Top-k: keep a fixed number

With top-k set to 3 at temperature 1, only chai, energy and joy stay in the raffle. Their shares are then rescaled so they add up to 100%: chai about 55%, energy 27% and joy 18%. Masala and sunshine cannot win at all.

Top-k was used in a well-known 2018 paper on story writing. The authors picked each word at random from the 10 most likely candidates. They found this worked much better than beam search, a method that hunts for a highly likely whole sentence.

In their tests, beam search tended to give short, generic sentences and repeated common phrases from the training data.

The weakness of top-k is that the number never changes. Sometimes the model has many good options, and a small k throws good ones away. Sometimes it has one clear answer, and a fixed k still lets weak options into the raffle.

Top-p: keep enough to reach a share

Top-p, also called nucleus sampling, sorts the tokens from most to least likely and adds up their chances. It stops as soon as the running total reaches p. Only the tokens it kept stay in the raffle, and their shares are rescaled to 100%.

The diagram walks through top-p 0.9 on the chai stall's list at temperature 1. Follow the loop: it keeps adding tokens until the total reaches 90%, then it draws.

Now look at the support bot copying order number 48213. Say that after the digits 4821, the model gives the digit 3 a 97% chance. Top-p 0.9 keeps just that one digit, so the copy comes out right.

Top-k 3 would still keep two wrong digits in the raffle, with a small chance of breaking the order number.

That is the main idea of the 2019 nucleus sampling paper. The kept group grows when the model is unsure, like choosing a tagline word. It shrinks when the model is sure, like copying a digit. The paper reports more varied text without losing fluency.

They work as a chain on every token. In vLLM's order, temperature reshapes the percentages, top-k and top-p cut the list, and one token is drawn from what is left. Temperature 0 skips the chain and takes the top token.

The order matters because top-p looks at percentages after temperature has reshaped them. At temperature 1, top-p 0.9 kept four tokens. At temperature 0.5, chai and energy make about 89.7%, just under 90%, so joy joins them and masala is dropped.

Here is the chain as two functions. It reuses to_probs from the earlier block, so run both blocks in one file. random.Random(seed) makes a random number generator, and its choices method draws one item using weights.

import random

def keep_top(probs, top_k=0, top_p=1.0):
    ranked = sorted(probs.items(), key=lambda item: item[1], reverse=True)
    if top_k > 0:
        ranked = ranked[:top_k]                      # keep only the k best
        total = sum(p for _, p in ranked)
        ranked = [(w, p / total) for w, p in ranked]  # rescale to add up to 1
    kept, running = [], 0.0
    for word, p in ranked:
        kept.append((word, p))
        running += p
        if running >= top_p:                         # stop once the total reaches top_p
            break
    return kept

def pick_next(scores, temperature=1.0, top_k=0, top_p=1.0, rng=random):
    if temperature == 0:
        return max(scores, key=scores.get)           # greedy: always the top token
    kept = keep_top(to_probs(scores, temperature), top_k, top_p)
    words = [w for w, _ in kept]
    weights = [p for _, p in kept]                   # choices() rescales for us
    return rng.choices(words, weights=weights)[0]

print([w for w, _ in keep_top(to_probs(scores, 1.0), top_p=0.9)])
print([w for w, _ in keep_top(to_probs(scores, 0.5), top_p=0.9)])

def ten_picks(seed):
    rng = random.Random(seed)                        # one generator for the whole run
    return [pick_next(scores, 1.0, top_p=0.9, rng=rng) for _ in range(10)]

print(ten_picks(7) == ten_picks(7), ten_picks(7))

Run it, and you should see these three lines (checked on Python 3.13; the exact picks in the last line could differ on another Python version):

['chai', 'energy', 'joy', 'masala']
['chai', 'energy', 'joy']
True ['chai', 'chai', 'energy', 'chai', 'energy', 'chai', 'chai', 'energy', 'chai', 'chai']

Two details in this code match the PyTorch version of vLLM 0.32.0's top-k and top-p code. After the top-k cut, the kept shares are rescaled before top-p adds them up. And random.choices treats the weights as relative, so the kept chances do not need to add up to 1.

Notice that the seed goes into one generator for the whole run. If you made a new generator with the same seed for every token, every token would get the same random number, which is not what you want.

A seed helps, but it cannot promise identical output from a hosted model, meaning one that runs on a model service's servers instead of your machine. A seed is a starting number for a random number generator.

The same seed gives the same sequence of random picks, as long as everything else stays the same. On your own laptop this works well. The Python docs say reusing a seed should reproduce the same sequence from run to run, as long as multiple threads are not running.

A model server has many more moving parts than your laptop. The diagram shows five reasons one prompt can come back different. Only the first two are fully under your control.

Batching means a server processes many users' requests together to use its hardware well. Which batch you land in can change the order in which the server adds up numbers.

Computers round decimals, so the order of additions can change the last digit. You can see this in Python: (0.1 + 0.2) + 0.3 prints 0.6000000000000001, while 0.1 + (0.2 + 0.3) prints 0.6.

Usually that difference is harmless. But when two tokens are almost tied, it can flip which one wins, and every later token can then differ.

A 2025 study traced this to rounding in floating-point maths, the way computers store decimals. Changing the batch size, the number of GPUs or the GPU type changed answers even with greedy decoding. A GPU is the graphics chip that does a model's maths.

In tests that stored numbers in a 16-bit format, one reasoning model's accuracy moved by up to 9% between such setups. A 2024 study ran five models ten times on eight tasks, each set up to be deterministic, meaning the same input should give the same output. Accuracy still changed by up to 15% between runs.

Those authors shared early results with insiders. Their abstract says the variation is perhaps essential to using hardware efficiently, with data mixed together in shared input buffers. It is offered as a likely cause, not a proven one.

So treat repeatability as something you design around, not something you get for free. The support bot now saves the prompt, settings, seed, model version and output for every extraction call. If a customer questions a record, the team can show exactly what the model said.

Use temperature 0 for jobs with one right answer, like extraction and labels. For creative writing, a sensible starting point is around 0.8 to 1.0, with top-p near 0.95. Treat all of these as starting points to test, not fixed rules.

The question to ask is simple: would two different answers both be fine? For a tagline, yes, and variety is the whole point. For an order number, no, and any variety is a bug. Here is how the team sets up each job.

The chart below turns that question into a quick decision. Every path ends with testing on your own prompts.

For extraction, temperature 0 is only half the fix. The team still checks every record in code. The order number must be five digits and must exist in the order system. Anything else goes back to the model once, then to a person.

Watch for these signs that a setting is off:

• Too high: facts drift, formats break and odd words appear, like the broken order records with "probably" in them.
• Too low: every new attempt looks the same, and long replies start repeating phrases, like the five near-identical taglines.
• About right: repeated extraction runs agree, and repeated tagline runs give different ideas that all stay on the brief.

Run the same saved prompts several times at each setting, and score the results with a check that fits the job. One run tells you almost nothing, because sampling is random by design.

Here is the process the team follows.

1. 1

   Collect real prompts

   Save 50 real support messages and 20 tagline briefs, with customer names and phone numbers removed.

2. 2

   Pick a score for each job

   For extraction, count how many records match the right answer. For taglines, count distinct ideas, and how many the copywriters would actually use.

3. 3

   Run each prompt several times

   Run every prompt five times per setting. This shows how much the answers wobble, not just whether one lucky run looked good.

4. 4

   Change one setting at a time

   Try temperature 0, 0.3 and 0.7 with top-p fixed. Then keep the best temperature and try a few top-p values.

5. 5

   Save everything

   Store the prompt, settings, seed, model version and output for every run, so any result can be traced later.

6. 6

   Test again after changes

   Rerun the set whenever the prompt, the model or the serving engine changes. Good settings for one model may not suit the next one.

In the team's test (made up), extraction at temperature 0 matched the right record 98 times in 100, against 94 at temperature 0.7. For the chai stall, temperature 0 gave 3 distinct taglines from 20 requests.

Temperature 1.0 with top-p 0.95 gave 17 distinct ones (made up), and the copywriters marked 6 as good enough to show the client. So the team now keeps two sets of settings, one per job.

Because they apply the settings in a different order. vLLM 0.32.0 divides by temperature first and then cuts with top-k and top-p. llama.cpp's default chain cuts first and applies temperature last, as of 19 October 2026.

vLLM is an open-source engine for serving models to many users at once, usually on GPUs. llama.cpp is an open-source engine that runs models on laptops, servers and GPUs, and its own server is called llama-server. Both are common ways to run a model you host yourself.

vLLM's sampler applies temperature, then min-p, then top-k and top-p together, then draws. Min-p is another cut: it drops tokens whose chance is below a set fraction of the top token's chance.

llama.cpp's default chain is penalties, DRY, top-n-sigma, top-k, typical-p, top-p, min-p, XTC and finally temperature. A penalty lowers the odds of tokens that have already appeared, to cut repetition. The penalties and the other extra names are not covered here, and they are switched off by default. Top-k, top-p, min-p and temperature are on.

Here is the same comparison in Python, reusing the two earlier blocks. Each function keeps the tokens its engine would keep, then rescales their chances at the chosen temperature.

def heat_then_cut(scores, temperature, top_p):   # vLLM order
    kept = keep_top(to_probs(scores, temperature), top_p=top_p)
    return to_probs({w: scores[w] for w, _ in kept}, temperature)

def cut_then_heat(scores, temperature, top_p):   # llama.cpp default order
    kept = keep_top(to_probs(scores, 1.0), top_p=top_p)
    return to_probs({w: scores[w] for w, _ in kept}, temperature)

for order in (heat_then_cut, cut_then_heat):
    probs = order(scores, temperature=0.5, top_p=0.9)
    print(order.__name__, {w: round(100 * p) for w, p in probs.items()})

Run it after the first two blocks, and you should see:

heat_then_cut {'chai': 74, 'energy': 18, 'joy': 8}
cut_then_heat {'chai': 72, 'energy': 18, 'joy': 8, 'masala': 2}

In vLLM's order, temperature 0.5 first pushes masala down to 2%, so top-p 0.9 drops it. In llama.cpp's order, top-p looks at the temperature 1 chances, where masala is 8%, so it survives. Temperature then shrinks it to 2%, but it can still win.

We ran the same case through llama.cpp's real sampler functions, from llama-cpp-python 0.3.36. The default order kept four tokens (chai 72.2%, energy 17.8%, joy 8.0%, masala 2.0%). Putting temperature first kept three. At temperature 1, both orders kept the same four.

So the same three numbers in a config file can mean different token sets on different engines. If you move a tuned setup from one engine to another, rerun your saved prompt tests. llama-server lets you change the order per request with its samplers field.

The built-in defaults differ too

When a request leaves a setting out, each engine fills it in. The two engines start from very different built-in values, checked in their source on 19 October 2026:

There is a catch. Both engines can also read default sampling settings from the model files themselves. vLLM's server loads the model's generation config file by default and uses it for any setting a request leaves out, including the penalties.

llama.cpp reads general.sampling keys from a model's GGUF file, its single-file model format, including the sampler order and the repeat penalty. A setting you pass yourself always wins, so send every setting you rely on in production.

On vLLM you can also start the server with --generation-config vllm. It then ignores the model's file and uses vLLM's own defaults.

Temperature 0 and very small temperatures

In vLLM 0.32.0, temperature 0 switches to greedy and resets top-p, top-k and min-p to off. Any temperature above 0 but below 0.01 is raised to 0.01 with a warning, so 0.001 still samples. Values above 2 are rejected.

That raise shows up in the source and in a log warning, not in the parameter's description, as of 19 October 2026, so pin your vLLM version. In llama.cpp, a temperature of 0 or below keeps only the top token, so the final draw is greedy.

You pin the seed, the engine version and the hardware, and you turn on the engine's repeatability mode. Even then, vLLM's docs promise repeatable results only on the same hardware and the same vLLM version.

vLLM's docs say it does not guarantee repeatable results by default, for the sake of speed. In offline use, setting VLLM_ENABLE_V1_MULTIPROCESSING=0 makes scheduling deterministic. In server mode, the only option is batch invariance.

Batch invariance means a request's output does not depend on which other requests share its batch. You switch it on with VLLM_BATCH_INVARIANT=1. As of 19 October 2026 the docs call it beta and list which GPUs it supports, so check your hardware first.

With batch invariance on, vLLM 0.32.0 warns when a sampled request has no seed of its own. So send a seed on every request you want to repeat, or use temperature 0.

llama-server picks a random seed when you send none, unless it was started with a fixed seed, so pass seed on each request. It also reuses cached work from earlier prompts by default, through its cache_prompt option.

Its README warns that this can make results nondeterministic, because logits are not guaranteed to be bit-for-bit identical across batch sizes. Turn cache_prompt off when you run a repeatability test.

1. 1

   Send every setting

   Pass temperature, top_p, top_k, min_p, the penalties and seed on each request. On llama-server also pass samplers, so no default from the engine or the model file sneaks in.

2. 2

   Pin versions

   Record the engine version, the model file and its checksum (a short fingerprint of the file), and the GPU type. A change to any of them can change outputs.

3. 3

   Switch on the engine's mode

   For a vLLM server, set VLLM_BATCH_INVARIANT=1 on supported hardware. For llama-server, set cache_prompt to false in repeatability tests.

4. 4

   Prove it with a rerun

   Send the same 20 saved prompts twice and diff the outputs. Expect zero differences; if some differ, log which settings or versions changed.

Explain it as two steps: the model scores every next token, and a decoding rule picks one. Then show that you match the rule to the job, know where your engine differs, and prove the choice with tests.

1. 1

   Describe the pick

   Say that the model turns scores into percentages for every token, and a decoding rule picks one token at a time.

2. 2

   Name the settings

   Greedy takes the top token. Temperature reshapes the odds. Top-k keeps a fixed number, and top-p keeps enough to reach a share.

3. 3

   Match settings to jobs

   Use temperature 0 for extraction and labels, and higher values for creative work, like the team's two jobs.

4. 4

   Admit the limits

   Say that temperature 0 and a seed reduce randomness but do not guarantee identical output, because batching and rounding can flip near ties.

5. 5

   Mention the engine

   Say that engines apply settings in different orders and with different defaults, so you send every setting explicitly and retest after moving engines.

6. 6

   Add checks, logs and tests

   Validate structured output in code, log prompt, settings, seed and versions, and rerun saved prompts after every model or engine update.