How do chat apps like ChatGPT work, in simple terms?

Underneath, a large language model reads your conversation and predicts the most likely next token, adds it, and repeats until the answer is done. Extra training on helpful conversations and human ratings turns that predictor into an assistant.

Why does AI give a different answer to the same question?

Apps usually sample the next token from the probabilities instead of always taking the top one. A different pick early in the answer changes everything after it. Lowering the temperature setting makes answers more repeatable.

Does an LLM plan its whole answer before writing?

It outputs one token at a time and cannot take back a token once it is written. Research suggests models do some looking ahead inside, for example when aiming for a rhyme, but every token is still chosen and committed one step at a time.

Does a language model understand what it says?

It has learned rich patterns about language and the world, which lets it explain, summarise and reason surprisingly well. But it has no way to check its words against reality by itself, so treat its answers as a smart draft, not as verified truth.

Module 2 · How a language model thinks · Lesson 6 of 32

Next-word prediction: the one trick behind chat AI

11 minFree GenAI course · BeginnerLesson 2.2By Saket Bhatnagar

Researched and drafted with AI, reviewed and edited by Saket Bhatnagar.

In short

A large language model writes by guessing the next token, adding it, and guessing again.
Each guess is a set of chances for every token it knows, learned from huge amounts of text.
Essays, code and poems are all built by this one loop, repeated hundreds of times.

Soon

Video lesson · coming soon

How LLMs predict the next word, explained in 3 minutes

Lala, your AI mentor, will walk you through this lesson in a short animated video. Until then, everything is here to read and try.

01 “Mere paas…” and the room finishes it
02 A fan who knows what comes next
03 Guess, add, repeat
04 You play the model
05 Learned from a giant guessing game
06 Likely is not the same as true
07 Build your own sentence

At a cousin’s engagement in Lucknow, Chachaji grabs the mic and says, in his deepest voice, “Mere paas…”. Before he can finish, half the room shouts “…maa hai!”

Nobody rehearsed that. Everyone has heard the line so many times that their brain predicted the ending on its own. Hold on to that moment. It is, quite honestly, the whole trick behind chat AI.

What does “predict the next word” mean?

You see a tiny version of this every day. Type “Happy” on your phone keyboard and it suggests “birthday”. It has learned what people usually type next.

“Next word” is a friendly shorthand. As you saw in the lesson on tokens, the model really predicts the next token, which is often a word and sometimes a piece of one. We’ll say “word” when it reads better.

Give a model “Mumbai is famous for its”, and it does not pick one answer straight away. It gives every token in its vocabulary a chance. “street” might get a big share, “beaches” and “local” smaller shares, and “giraffes” almost nothing. Those numbers are made up for this example, but the shape is real: a few likely words, and a long tail of unlikely ones.

Think of it like this

The radio commentator who has heard ten thousand overs

Kabir’s nana has listened to cricket commentary on the radio since the 1970s. When the commentator says “and that’s gone high in the air, the fielder is under it, and…”, Nana says “taken!” before the word arrives. He is not psychic. He has heard thousands of overs, and he knows what usually comes next. He is usually right. Now and then the ball is dropped and the sentence goes somewhere else, and that surprise is part of the fun.

In real lifeIn AI

Thousands of matches heard on the radio: In AI: Huge amounts of text read during training
What has happened in the over so far: In AI: The text so far: your prompt plus what the model has already written
Nana’s confident “taken!”: In AI: The token with the highest probability
The rare dropped catch: In AI: A less likely token that sometimes gets picked
Nana guessing one word at a time as the commentary flows: In AI: The model choosing one token at a time, never the whole answer in one go

The loop: guess, add, repeat

One guess gives you one token. To write a whole answer, the model runs a loop. It reads all the text so far, scores every possible next token, picks one, adds it to the end, and then reads everything again, now one token longer.

How an LLM writes an answer

Drag to explore

The model scores every possible next token, one is picked and added to the text, and the loop repeats until an end token is picked.

Start
The text so far is “My chai is too”. The model scores every token; “hot” comes out on top, with “sweet” close behind.
Add
“hot” is added. The text is now “My chai is too hot”, and the model reads the whole line again.
Guess again
This time “to”, a comma or “and” score well, because the sentence could go on in a few ways.
Keep going
Token by token the sentence grows: “My chai is too hot to drink right now.”
Stop
The model picks the special end token, and the app stops writing.

How does it know when to stop? During training, every document ends with a special end token, so the model learns to predict that too. Apps also set a maximum length as a safety net, so a reply cannot run on forever.

Try it: build a sentence one word at a time

Below you can play the model. Pick one of three starting lines. You’ll see the AI’s top guesses for the next piece as bars, each with its chance; the longer the bar, the higher the chance. Tap one to add it, and a fresh set of guesses appears for the next spot. In this demo each piece can be a short phrase, to keep the game quick; a real model goes one token at a time.

Try it yourselfYou are the language modelPick a starting line, tap one of the top guesses (or press “Let the AI pick”), and move the temperature slider to see the bars change.

Start with “My chai is too” and tap the longest bar, “hot”. Then start again and tap a shorter one, like “cold”. Same start, different story.
Press “Let the AI pick” a few times from the same start. It usually picks a likely piece, but not always.
Slide the temperature down to 0. The top bar takes everything, so the AI always picks the same word.
Slide it towards 1.5. The bars even out, and odd words get a real chance.

That “Let the AI pick” button is sampling. It is why the same question can get a slightly different answer each time. The slider is called temperature, and it gets a full lesson of its own later in this module. For now: low means safe and repeatable, high means adventurous.

Where do the probabilities come from?

Remember the guess, check and nudge loop from module 1, where a model learned to judge mangoes? Training a language model is the same loop with a different game. Hide the next token in a real sentence, let the model guess, check the real answer, and nudge its settings.

That game is played on a vast library of text: websites, books, articles and code, adding up to trillions of tokens. The text is its own answer key, so nobody has to label anything by hand. Researchers call this self-supervised learning.

Here is the surprising part. To get really good at guessing the next token, the model is pushed to pick up a lot along the way. To finish “The capital of India is”, it needs facts. To finish a sentence in Tamil-English mixed text, it needs both languages. To finish the last line of a detective story, it needs to follow the plot.

Your phone keyboard vs a large language model
	Phone keyboard	Large language model
Looks at	The last word or two	The whole conversation, often thousands of tokens
Learned from	Common phrases and your own typing	A huge share of public text, books and code
Settings inside	Small	Billions
What it does	Offers three words; you choose	Picks a token itself and keeps going until it stops

From autocomplete to a chat assistant

A model trained only on the guessing game is a brilliant autocomplete, but a poor helper. Ask it “What should I cook tonight?” and it may continue with more questions, as if it were writing a quiz page it once read.

To become an assistant, it is trained further on examples of helpful conversations, and people rate its answers. You’ll see how in the lesson on training. The engine underneath does not change: it is still predicting the next token.

In a chat app, your whole conversation is laid out as one long text, with hidden markers for who said what. Then the model continues the text from the point where the assistant’s turn begins. Your chat is, in a sense, one long sentence it keeps finishing.

Check yourself

An LLM is writing an answer. What happens right after it picks a token?

Pick an answer to see why it is right or wrong.

Check yourself

Meera asks the same question twice and gets two slightly different answers. Why?

Pick an answer to see why it is right or wrong.

Check yourself

Where do the next-token probabilities come from?

Pick an answer to see why it is right or wrong.

Remember these three

An LLM writes by predicting the next token, adding it, and repeating until it picks an end token.
Each prediction is a set of probabilities learned from huge amounts of text; sampling picks one.
The model chooses what sounds likely, not what it has checked, so fluent does not always mean true.

Questions people ask

How do chat apps like ChatGPT work, in simple terms?: Underneath, a large language model reads your conversation and predicts the most likely next token, adds it, and repeats until the answer is done. Extra training on helpful conversations and human ratings turns that predictor into an assistant.
Why does AI give a different answer to the same question?: Apps usually sample the next token from the probabilities instead of always taking the top one. A different pick early in the answer changes everything after it. Lowering the temperature setting makes answers more repeatable.
Does an LLM plan its whole answer before writing?: It outputs one token at a time and cannot take back a token once it is written. Research suggests models do some looking ahead inside, for example when aiming for a rhyme, but every token is still chosen and committed one step at a time.
Does a language model understand what it says?: It has learned rich patterns about language and the world, which lets it explain, summarise and reason surprisingly well. But it has no way to check its words against reality by itself, so treat its answers as a smart draft, not as verified truth.