How language models are trained

Rohan grew up in Indore eating his mother’s poha every morning. He read every cookbook in the house and watched a thousand cooking videos. He knows food deeply. But on his first day at a restaurant, a customer says “one plate poha, less oil, extra sev” and Rohan freezes. Knowing food and serving a customer are two different skills.

A language model has the same problem, and it is solved the same way. First a lot of reading, then practice with real orders, then feedback from the people who taste the food. Let’s walk through all three.

In the last few lessons you saw how a model works: it cuts text into tokens, predicts the next one, and uses attention to decide which words matter. But a brand-new model knows nothing. Its billions of settings start as random numbers, so its first guesses are gibberish.

Training is the job of setting those numbers. It happens in three stages, and each one teaches something different. The first teaches language and knowledge. The second teaches manners and format. The third teaches judgement: which of two answers people actually prefer.

Pretraining starts with text. A lot of it: public web pages, books, encyclopedias, news, forums and computer code. Teams clean it first, removing spam, copies and harmful pages. For today’s large models, the pile comes to trillions of tokens. Reading one book a day, a person would need many thousands of lifetimes to get through it.

The task is the one you already know from the next-word lesson. Hide the next token, let the model guess, check the real answer, and nudge the numbers so the right token becomes a little more likely. It is the guess, check and nudge loop from module 1, repeated trillions of times.

Nobody writes labels for this. The text is its own answer key, because the next word is always sitting right there. That is why it can scale to so much data. Along the way, the model picks up grammar, facts, styles, a few dozen languages, and even some reasoning, because all of those help it guess better.

This stage is huge. It runs for weeks or months on thousands of specialised chips, and for the biggest models the bill can run to hundreds of crores of rupees or more. Only a handful of companies and labs do it at the top end.

Here is the surprise. Ask a base model “What is the capital of Rajasthan?” and it may reply with more quiz questions: “What is the capital of Gujarat? What is the capital of Punjab?” To it, your question looks like the start of a quiz sheet, and continuing the sheet is the most likely next text. It knows the answer is Jaipur, but it does not know it is supposed to answer you.

Now the apprenticeship. The team gathers example conversations: a question or task, followed by a good answer. Some are written by trained people, some are checked by them, and these days many are drafted by other models and then filtered. The count is far smaller than pretraining, roughly tens of thousands to a few million examples.

The model trains on these the same way, guessing the next token, but now the text is always “question, then helpful answer”. After enough of them, the pattern sinks in. When it sees a question, the most likely next text becomes an answer, in a polite and clear format. This kind is often called instruction tuning or supervised fine-tuning.

Fine-tuning mostly teaches habits, not new knowledge. It shapes tone, format, when to use bullet points, when to ask a follow-up question, and when to refuse something harmful. The knowledge still comes from the library read in stage one.

Writing a perfect answer is hard. Picking the better of two answers is much simpler. You may not be able to cook a perfect biryani, but you can taste two plates and say which one wins. Stage three uses exactly that.

The model writes two or more answers to the same question. People compare them and mark the one they prefer: more correct, clearer, kinder, safer. Their choices train a separate scorer that learns to predict what people like. Then the main model practises against that scorer, and it is nudged towards answers that score well.

There are newer variations. Some methods skip the separate scorer and learn straight from the pairs of preferred and rejected answers. Some use another model as the judge, guided by a written list of rules. And for maths and code, an answer can often be checked automatically: the sum is right or wrong, the code passes its tests or fails. Training on those checks is a big reason recent models reason better, and it is how many “thinking” models are made.

Three things follow from this. First, the model’s knowledge stops at the date its library was collected, called the knowledge cutoff. Anything after that, it has never read. Second, training is finished before you ever type a word. Your chat does not teach the model anything; it only uses what is already in its numbers.

Third, the model learned to sound like a good answer. Most of the time, sounding right and being right go together. Sometimes they do not, and that gap is where made-up answers come from. Hold that thought until the lesson on hallucinations.