How LLMs Work: The Life of a Prompt, Step by Step

It invents answers that look right. An LLM writes the text most likely to come next, not text it has checked. So a question about today's price can get a confident, invented number. Knowing how LLMs work explains why, and shows what your app must do instead.

Picture a common setup. A recipe-helper app in Kolkata lets users type what is in their kitchen, and it suggests a dish. The setup is made up, but the problems in it are real. One user typed: "I have leftover rice and two eggs. What can I cook with them in 15 minutes?" The app replied with a neat recipe for egg fried rice.

Two other messages puzzle the team. A user asked for today's hilsa price at a Kolkata fish market, and the app confidently said Rs 1,400 per kg. Nobody ever gave the app a price list. Another user wrote that the app somehow remembers she is vegetarian.

This guide follows the rice and eggs prompt through the model, and both puzzles get solved on the way. A few numbers set the scene. The app answers about 20,000 questions a day, and each question and answer together use about 700 tokens.

The team pays a model provider, a company that runs the model and bills for its use. That comes to about 8 paise (0.08 rupees) per answer, roughly Rs 1,600 a day. Tokens, the unit behind that bill, are explained below.

You need five: LLM, token, parameters, inference and context window. Each gets a plain meaning here, and the rest of the guide uses each one the same way.

LLM. A large language model is a program trained on a huge amount of text to do one job: guess the next small piece of text. "Large" means it holds billions of internal numbers. One 2020 model described in its paper had 175 billion.

Token. A token is the small piece of text the model reads and writes. It can be a whole common word like "rice", part of a longer word, or a punctuation mark. Whether "leftover" is one token or two depends on the tokenizer, the program that cuts text up.

Parameters. Parameters, also called weights, are the internal numbers set during training. They hold patterns of language, not a neat list of facts.

Inference. Inference is the model answering a request with its finished parameters. The parameters do not change during inference, so the model does not learn from your question.

Context window. The context window is the most tokens the model can handle in one request, counting both the prompt and the answer. Think of it as the one page the model can see.

LLMs work in a loop: they turn your prompt into numbers, score every possible next token, pick one, add it to the text and repeat. The scoring happens inside a stack of layers. Engineers call this autoregressive generation: each output becomes part of the next input.

Your phone keyboard runs a tiny version of this. As you type, it suggests a next word, you tap one, and it suggests again. An LLM runs the same suggest-and-pick loop, but it picks each token itself.

Where the picture breaks: a keyboard suggestion is usually a quick guess from your last few words. An LLM reads every token on its page through many layers before each guess. That is why it can follow instructions, not just finish a word.

Answer: about 141 times. The model runs once for each of the roughly 140 recipe tokens, then once more to pick a special end token. The arrow marked No is why answers appear word by word on your screen.

In two steps. A tokenizer cuts the text into tokens and swaps each one for an ID number. Then the model swaps each ID for an embedding, a long list of numbers learned during training.

Step 1: tokens and token IDs

A tokenizer has a fixed vocabulary, and each token in it has its own ID number. The 2017 paper that introduced the transformer used one shared vocabulary of about 37,000 tokens for English and German. Vocabulary sizes differ between models, so check yours.

The rice and eggs prompt has 16 words and 2 punctuation marks, so expect about 18 tokens. Common words like "rice" are often one token, and "15" may be one or two. Say "rice" became ID 19204 (made up). From here on, the model sees only these numbers.

Many tokenizers build their vocabulary with byte pair encoding. It starts from single characters and keeps merging the pair that appears together most often, for a chosen number of merges. A 2016 translation paper used it so rare words split into familiar pieces instead of becoming unknown.

Step 2: embeddings

An embedding is a list of numbers that stands for a token. The model keeps a table with one row per token ID and looks up the row for each ID in the prompt. The prompt is now a stack of number lists.

Rows are long. The 2017 paper's base model used 512 numbers per token, and one large 2020 model used 12,288. The rows are learned: training adjusts them along with every other parameter.

The model also needs each token's position. Without it, "rice before eggs" and "eggs before rice" would look the same. The 2017 paper added wave-shaped position signals to the embeddings; a 2021 paper instead rotated, by position, the numbers that attention compares. That method is called rotary position embedding.

It refines every token's numbers, layer by layer. The transformer is a stack of layers with the same shape but their own weights: 6 per stack in the 2017 paper, 96 in one large 2020 model.

Each layer does two jobs in order. First comes attention, where each token gathers meaning from earlier tokens. Then comes a feed-forward step, where each token's numbers go through the same small calculation on their own.

Attention is easiest to see on one word. In the prompt, "them" means nothing by itself. Attention lets it score every earlier token for relevance and mix in the most relevant ones, here "rice" and "eggs".

In the diagram below, follow the arrows into "them". Most meaning flows in from "rice" and "eggs", a little from "leftover" and almost nothing from "cook". Real models learn their own patterns, and those are rarely this clean.

Notice what is missing. "Minutes" comes after "them", so "them" cannot look at it. In most LLMs, which write text left to right using only a decoder (the part that writes output), each token attends only to itself and earlier tokens. The 2017 paper did this in its decoder with a mask that blocks later positions.

Each layer runs several attention heads side by side; a head is one separate attention pattern. The 2017 paper's base model used 8 heads per layer. Its authors showed two heads that apparently linked pronouns back to the words they refer to.

By the final layer, the numbers for the last token are used to score every possible next token. That score list is where the next section starts.

It scores every token in its vocabulary, turns the scores into probabilities with a step called softmax, and picks one. Then it adds that token and goes round again.

Say the model has written "You can make egg" and must choose the next token. The raw scores are called logits: plain numbers where bigger means more likely. Softmax raises e to the power of each score and divides by the total, so the results add up to 100 percent.

Temperature is a setting that divides every logit before softmax. Below 1 it sharpens the choice, so fried jumps to 80.4 percent. Above 1 it flattens the choice, so even cake rises to 3.2 percent.

Here is the table as NumPy code you can run. NumPy is the standard Python library for arrays of numbers.

import numpy as np

tokens = ["fried", "bhurji", "curry", "pulao", "cake"]
logits = np.array([4.0, 3.1, 2.6, 2.0, -1.0])

def softmax(scores, temperature=1.0):
    e = np.exp(scores / temperature)
    return e / e.sum()

for t in (0.5, 1.0, 2.0):
    probs = softmax(logits, t)
    print(t, [f"{p:.1%}" for p in probs])

rng = np.random.default_rng()
print("Picked:", rng.choice(tokens, p=softmax(logits)))

What just happened? The softmax function turned five scores into percentages, once per temperature. The first line should read 0.5 ['80.4%', '13.3%', '4.9%', '1.5%', '0.0%'], matching the table after rounding. The last line samples, so run it a few times: mostly fried, sometimes bhurji, rarely cake.

That weighted random pick is called sampling. Always taking the top token, called greedy decoding, sounds safer. But a 2020 study found that choosing text by likelihood makes open-ended writing bland and repetitive.

Defaults differ, so check yours. As of 19 October 2026, one widely used open-source generation library (version 5.20.2) uses greedy decoding unless you or the model's settings file switch sampling on. Its default temperature is 1.0.

It repeats the whole process for every new token. The token it just picked is added to the end of the input, and the longer input goes through the model again.

Here is how the reply to the rice and eggs prompt grows, pass by pass.

1. 1

   Pass 1

   The input is the prompt, about 18 tokens, plus the app's instructions. The model scores every token and picks "You".

2. 2

   Pass 2

   "You" is added to the input. The model runs again and picks "can".

3. 3

   Passes 3 and 4

   The model adds "make" and then "egg". The input grows by one token each time.

4. 4

   Pass 5

   This is the moment from the table. Fried has 55.7 percent at temperature 1, and the sampler picks it.

5. 5

   Passes 6 to 140

   The model writes the rest of the recipe, one token per pass, each time reading everything written so far.

6. 6

   Pass 141

   The sampler picks the special end token, which means the answer is complete. The loop stops.

The loop usually stops for one of these reasons:

• The model picks its special end token. It learned during training to produce this token where a finished answer ends.
• The answer reaches the maximum number of new tokens the app set, such as 400.
• The model writes a stop sequence, a short string the app said should end the answer.
• There is no room left in the context window for another token.

Here is the loop as Python you can run. The model is a toy stand-in: a fixed table of next-token chances that looks only at the last token. The loop around it has the same shape as a real one.

import numpy as np

VOCAB = ["?", "You", "can", "make", "egg", "fried", "rice", ".", "END"]
END_TOKEN = VOCAB.index("END")
MAX_NEW_TOKENS = 20

# Toy stand-in for a model: next-token chances that depend only on the
# last token. A real model reads every token in the window.
TABLE = {"?": {"You": 1.0}, "You": {"can": 1.0}, "can": {"make": 1.0},
         "make": {"egg": 1.0}, "egg": {"fried": 0.8, "rice": 0.2},
         "fried": {"rice": 1.0}, "rice": {".": 1.0}, ".": {"END": 1.0}}

def run_model(token_ids):
    probs = np.zeros(len(VOCAB))
    for word, p in TABLE[VOCAB[token_ids[-1]]].items():
        probs[VOCAB.index(word)] = p
    return probs

def answer(prompt_ids, rng):
    token_ids = list(prompt_ids)
    new_ids = []
    for _ in range(MAX_NEW_TOKENS):            # hard limit on answer length
        probs = run_model(token_ids)            # one pass through the model
        next_id = int(rng.choice(len(VOCAB), p=probs))  # sample one ID
        if next_id == END_TOKEN:                # the model says it is done
            break
        token_ids.append(next_id)               # the new token becomes input
        new_ids.append(next_id)
    return " ".join(VOCAB[i] for i in new_ids)

rng = np.random.default_rng(11)
prompt_ids = [VOCAB.index("?")]   # stands in for the whole tokenized prompt
for _ in range(3):
    print(answer(prompt_ids, rng))

What just happened? Each call ran the loop until the toy model picked END. With seed 11 on NumPy 2.5.4, the three lines print "You can make egg fried rice ." twice, then "You can make egg rice ." once. The 20 percent branch at "egg" is sampling at work.

The key line is token_ids.append(next_id). It feeds each new token back in, so the next pass reads one more token than the last. MAX_NEW_TOKENS is the length limit, so a confused model cannot write forever.

Chat apps usually show each token as soon as it is picked, instead of waiting for the loop to finish. This is called streaming, and it is why the recipe appears a few words at a time.

It is the most tokens a model can handle in one request. It counts everything: the app's instructions, the chat history, the new message and the answer the model writes.

Sizes differ a lot between models. The large 2020 model mentioned earlier had a window of 2,048 tokens. Read the exact limit on your model's own documentation rather than a number from a blog, including this one.

In a chat, every earlier message is sent again with each new one, so the total keeps growing. When it no longer fits, the app must drop or shorten something. Common choices are trimming the oldest messages or replacing them with a short summary.

Picture the window as one page. The model reads that page and nothing else, and its answer must fit on the same page. Where the picture breaks: a careful person reads a page evenly, and a model may not.

A 2023 study tested models on long inputs. They used facts at the start or end of the context more reliably than facts in the middle, even models built for long contexts. Longer prompts also cost more tokens.

No, a plain LLM does neither. It predicts tokens from its fixed parameters and from whatever is inside the current context window. Anything that looks like search or memory is the app's work.

Why the model invented a hilsa price

Now the first puzzle makes sense. The model did not open a website. It produced tokens that look like a typical answer to that kind of question, and such answers usually contain a price per kg.

Its knowledge stops at its knowledge cutoff, the date its training text was collected. When a model states something false with confidence, people call it a hallucination. The model is not lying; likely text and true text have come apart.

The fix lives in the app. If the app wants real prices, its code must fetch them from a trusted price feed and paste them into the prompt. A 2020 paper on retrieval-augmented generation notes that knowledge stored in parameters is hard to update or trace to a source. Its system also fetches relevant passages at question time and adds them to what the model reads.

Why the app seemed to remember the vegetarian user

The app saves a short note for each user in its own database, such as "vegetarian, no onion or garlic". Before every request, its server pastes that note into the prompt. The model is not remembering her; the note is simply on the page every time.

The diagram below shows what one request contains. Count the boxes that feed the app's server before anything reaches the model.

Two myths come from the same mix-up:

• Myth: the model learns from your chat as you talk. Reality: its parameters do not change during inference. A provider may use stored chats to train future versions, depending on its policy.
• Myth: the model stores whole documents and copies them out. Reality: it mostly stores patterns, but it can repeat some training text word for word. A 2022 study found this grows with model size and with how often the text was duplicated.

In each attention head, every token makes three vectors, or lists of numbers. They are a query (what it looks for), a key (what it offers) and a value (what it passes on). Scores are queries times keys, divided by the square root of the vector size. A mask blocks later tokens, and softmax turns each row into weights.

"Causal" means each token sees only itself and the past. The code uses toy sizes, 5 tokens with 8 numbers each, and random weights instead of trained ones.

import numpy as np

rng = np.random.default_rng(1)
n, d = 5, 8                       # 5 tokens, 8 numbers each (toy sizes)
x = rng.normal(size=(n, d))       # token vectors entering one layer
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))
q, k, v = x @ Wq, x @ Wk, x @ Wv  # queries, keys, values

scores = q @ k.T / np.sqrt(d)     # how well each query matches each key
future = np.triu(np.ones((n, n), dtype=bool), k=1)
scores[future] = -np.inf          # block later tokens: the causal mask

weights = np.exp(scores - scores.max(axis=1, keepdims=True))
weights /= weights.sum(axis=1, keepdims=True)
out = weights @ v                 # each token's new numbers

print(np.round(weights, 2))
print("rows sum to 1:", np.allclose(weights.sum(axis=1), 1))

What just happened? The printed weights form a triangle: row 1 is 1 followed by four zeros, and every number above the diagonal is 0. Each row sums to 1, so the last line prints True. The tests for this guide also changed the fifth token and confirmed the first four outputs did not move.

That last property matters. In a causal model, an earlier token's keys and values never depend on later tokens. The next section turns that fact into a large saving.

Because recomputing the past is wasted work. In a causal model the keys and values of earlier tokens never change, so servers compute them once, store them and reuse them on every later pass. That store is the key-value cache, or KV cache.

Reading the whole prompt in the first pass is called prefill. Each later pass, called decode, handles one new token and reads the cached keys and values. The code below checks that the cache changes the work, not the answer.

import numpy as np

rng = np.random.default_rng(2)
d = 8
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))
tokens = rng.normal(size=(6, d))  # 6 token vectors arriving one by one

def attend(q, K, V):
    s = K @ q / np.sqrt(d)
    w = np.exp(s - s.max())
    return (w / w.sum()) @ V

# Without a cache: recompute every key and value on every pass
slow, rows = [], 0
for t in range(len(tokens)):
    K, V = tokens[:t + 1] @ Wk, tokens[:t + 1] @ Wv
    rows += t + 1
    slow.append(attend(tokens[t] @ Wq, K, V))

# With a cache: compute each token's key and value once, then keep them
K_cache, V_cache, fast = [], [], []
for t in range(len(tokens)):
    K_cache.append(tokens[t] @ Wk)
    V_cache.append(tokens[t] @ Wv)
    fast.append(attend(tokens[t] @ Wq, np.array(K_cache), np.array(V_cache)))

print("same output:", np.allclose(slow, fast))
print("key and value rows computed:", rows, "without cache,", len(tokens), "with cache")

What just happened? Both loops produce the same outputs, so the first line prints same output: True. Without the cache the loop computed 21 key and value rows for 6 tokens; with it, 6. The uncached total grows with the square of the length.

The open-source generation library mentioned earlier turns this cache on by default, as of version 5.20.2.

The memory bill

The cache costs memory. For each token it holds one key and one value per layer, each as long as the model's width. For the large 2020 model that is 2 times 96 layers times 12,288 numbers.

At 2 bytes per number, that is 4,718,592 bytes, exactly 4.5 MiB per token. A full 2,048-token window then needs about 9.66 GB (9.0 GiB) for one request. That memory usually sits on the GPU, the chip that runs the model, beside the parameters.

Those numbers assume every layer attends to the whole window. The 2020 paper itself alternated full layers with local ones, which can cache less. They also assume every head keeps its own keys and values. A 2023 paper describes grouped-query attention, where several query heads share one set of keys and values, so the cache holds fewer of them.

Mostly because tokenizers learn their vocabulary from training text, and English usually dominates that text. Scripts that appear less get fewer vocabulary entries, so their words split into more pieces.

Some tokenizers also start from raw UTF-8 bytes, the way computers store text. English letters take 1 byte each, while Devanagari and Bengali characters take 3. With few learned merges, one Hindi or Bengali character can cost more than one token. A 2026 study proposes adding such multi-token characters to the vocabulary as single tokens.

for word in ["rice", "चावल", "ভাত"]:
    print(word, len(word), "characters,", len(word.encode("utf-8")), "bytes")

Expected output: rice is 4 characters and 4 bytes, the Hindi word for rice is 4 characters and 12 bytes, and the Bengali word is 3 characters and 9 bytes. Bytes are not tokens, but they show how much more raw material each word gives the tokenizer.

The same study measured ten popular tokenizers on identical sentences. Averaged over 997 sentence pairs, Hindi needed 1.37 to 7.51 times the tokens of English. Bengali needed 1.39 to 9.70, depending on the tokenizer.

For the app, a Bengali user's chat costs more, fills the window sooner and takes more passes to write. Measure it: run 100 real user messages per language through your model's own tokenizer and compare the totals.

Explain it as a loop, then show what the system around the model must provide.

1. 1

   Give the one-line loop

   Say that an LLM predicts the next token, adds it to the input, and repeats until an end token or a length limit.

2. 2

   Walk from text to numbers

   Mention the tokenizer, token IDs and embeddings. Add that cost and limits are counted in tokens, not words, and differ by language.

3. 3

   Name attention in one sentence

   Each token borrows meaning from earlier tokens, layer after layer. Give a small example, like "them" linking back to "rice".

4. 4

   Explain probabilities and sampling

   Say the model outputs a probability for every token and the app samples one. This is why the same prompt can give different answers.

5. 5

   Bring in the context window

   Everything the model knows about this request must fit in one window. Long chats need trimming or summaries.

6. 6

   Do the memory maths

   Per token, the KV cache holds a key and a value per layer. Multiply by the window and by concurrent users to size GPU memory.

7. 7

   Say what the app must add

   Fresh facts and memory come from your code, not the model. So do limits like maximum length.

Rough numbers make your answer stronger. About 700 tokens per question times 20,000 questions is 14 million tokens a day, roughly Rs 1,600. Then add that longer chats raise the bill, because every message is sent again each turn.