How LLMs Are Trained: Pretraining, Fine-Tuning and RLHF

It learns to continue text, not to answer it. Type a question into a model trained only on web pages, and it may reply with three more questions. That is why understanding how LLMs are trained means understanding several stages, not one.

Picture a common setup. A farmers' helpline gets about 3,000 calls a day in Kannada, Tamil and Hindi, and most questions repeat. A small team wants a model that answers them in the caller's own language. The model will cover seven languages: the callers' three, English, and Telugu, Malayalam and Marathi, so the helpline can grow into nearby states.

The team has a budget of Rs 1 crore (10 million rupees), and decides to train its own small model. The setup is made up, but the stages are the real ones. Starting from scratch is rarely the cheapest choice, but it lets you see every stage.

You need six words: token, parameter, loss, base model, fine-tuning and checkpoint. Each one gets a plain meaning here, with the helpline model as the example.

Token. A token is a small piece of text the model reads and writes: a whole word, part of a word or a few letters. A tokenizer is the tool that does the splitting. In two public tokenizers we tested, the English word farmer is one token. The Kannada word for sowing takes 5 tokens in one and 14 in the other.

Parameter. A parameter is one number inside the model, and large models have billions of them. Training means adjusting these numbers until the model's guesses get better. The helpline model has 2 billion parameters.

Loss. Loss is a score for how wrong the model's guess was, and lower is better. If the model is confident about the right next token, its loss is small. If it bets on the wrong token, its loss is large.

Base model. A base model is what you get after the first big training stage. It is good at continuing text, but it does not yet treat your text as a request to answer.

Fine-tuning. Fine-tuning means training an already-trained model a little more on a smaller, focused set of examples. Example: teaching the helpline model to answer politely, in the caller's own language.

Checkpoint. A checkpoint is a saved copy of all the model's parameters at one moment in training. If a later stage makes things worse, the team goes back to the last good checkpoint.

LLMs are trained in stages: collect and clean text, pretrain on next-token prediction, then fine-tune for instructions, preferences and safety. In the helpline team's plan, tests run between stages, and a checkpoint moves on only when it passes them.

Notice in the diagram that only pretraining reads the giant pile of text. The stages after it use much smaller sets of examples, written or judged by people.

One picture helps across all the stages: a new helpline worker. Each part of the worker's training maps to one stage:

• Reading every farming book in the library is pretraining: lots of knowledge, no idea how to take a call.
• A training week of listening to good recorded calls is instruction tuning.
• A senior who hears two replies and says which was better is preference tuning.
• Practice calls with angry or tricky callers are safety tuning and red-teaming.

Where the picture breaks: a worker can learn a rule from one sentence and understand why. A model only shifts its odds, so it needs many examples, and it can still slip on wording it never saw.

Training data mostly comes from public web pages, plus books, news, code and other text a team is allowed to use. Before training, teams remove junk, copies and personal data, because the model learns from whatever it reads.

The team starts with Common Crawl, a free, open archive of web pages collected since 2007. As of October 2026, its site says it holds over 300 billion pages. The team adds openly licensed books, government documents and news text it may use, about 600 billion tokens in all.

Most of that pile is not worth reading. It has cookie banners, menus, spam, machine-made gibberish and the same news story copied across hundreds of sites. These are the main cleaning steps:

• Language check: keep only the seven languages the helpline model needs, and drop pages with broken script.
• Quality filter: a small helper model scores each page and drops low scorers. In one 2024 study, keeping only educational pages clearly raised scores on knowledge and reasoning tests.
• Deduplication: remove exact copies and near-copies. In a 2021 study, models trained on deduplicated data repeated memorised training text about ten times less often in its tests.
• Personal data: mask phone numbers, emails and ID numbers, so the model is less likely to learn and repeat them.
• Test-set removal: delete any text that also appears in your later tests, so scores stay honest.

After cleaning, about 200 billion tokens remain. In the team's pile, Kannada and Tamil are a small share next to English, so the team sets the data mix: the share of each language in training. It includes the best Kannada and Tamil text twice, so English does not drown it out.

Pretraining is the first and biggest training stage: the model reads billions of tokens and keeps guessing the next one. To guess well, it has to pick up grammar, facts and common patterns in how people reason in text.

The text itself holds the right answer, so no person has to label anything. This is called self-supervised learning. Here is one round: the model sees Bengaluru is the capital of, and must guess the next token.

Early in training, it gives Karnataka a probability of only 0.20, meaning 20% of its bet. Treat Karnataka as one token here for simplicity. The loss is the negative logarithm of that probability, a maths step that turns a confident right guess into a small number.

If you code in Python, you can check these numbers yourself. If you do not, skip the code: the two lines of output are the point.

import math

def loss(p_correct):
    # p_correct: the probability the model gave the right next token
    return -math.log(p_correct)

print(round(loss(0.20), 2))   # early in training
print(round(loss(0.90), 2))   # later in training

It prints 1.61 and then 0.11. So moving the right answer from 20% to 90% cuts the loss from 1.61 to 0.11.

Training then uses gradient descent: a method that works out which way to nudge each parameter so the loss drops a little. The model guesses every token in a batch, a group of text pieces processed together, and the losses are averaged. Then every parameter gets one small nudge, and the next batch starts.

The helpline model reads its 200 billion tokens on 32 GPUs in about a week. A GPU, or graphics processing unit, is a chip that does huge numbers of small sums at the same time. Spreading one run across many GPUs is called distributed training.

How big should the model be? A 2022 study found that, for a fixed amount of computing power, model size and training text should grow in equal proportion. Its tables work out to roughly 20 training tokens per parameter.

A 2023 study added the cost of answering questions afterwards. When a model will serve very many requests, around a billion in its analysis, it found a smaller model trained on more text is the better deal. The helpline, at about a million calls a year, is far below that scale. It picks a small model so one low-cost GPU can answer each call quickly.

What does a base model do after pretraining?

Pause and predict: someone types a Kannada question asking when to sow ragi in Mandya. What does the base model send back?

Often, it continues the text instead of answering it, for example with more questions in the same style. The 2022 instruction study names the gap: predicting the next token on a web page is a different goal from following a request. The knowledge is in there. Type the start of an answer, The best time to sow ragi in Mandya is, and it may finish it well.

What it lacks is the habit of treating text as a request. The next two stages build that habit.

Supervised fine-tuning, also called instruction tuning, trains the base model on example conversations where a person wrote the ideal answer. Supervised means each example comes with the right answer, written by a person.

The team hires 40 native speakers to write 20,000 examples across the seven languages. Each example is a pair: a prompt and the answer the team wants. One Kannada prompt asks when to sow ragi in Mandya. The written answer gives the usual sowing months and suggests checking with the local agriculture office.

The training method barely changes: it is still next-token prediction, now on these pairs. A common choice is to count only the answer part towards the loss. Then the model learns how to answer, not how to write questions.

The dataset is tiny next to pretraining: thousands of examples, not billions of tokens. Yet this stage changes a lot. In a 2022 study, a 1.3 billion parameter model tuned this way and then with preferences beat an untuned 175 billion parameter model. The study's raters preferred its answers on held-out prompts from the study's own users.

RLHF, short for reinforcement learning from human feedback, teaches a model from people's choices between answers, using a separate scoring model. DPO, short for direct preference optimization, learns from the same choices directly, with no separate scoring model and no trial-and-error loop.

Choices help because judging is easier than writing. A rater can compare many pairs of answers in the time it takes to write one perfect answer.

The instruction-tuned helpline model writes two answers to each of 30,000 prompts. Native-speaker raters pick the better answer in each pair, at Rs 40 per choice (made up). Each result is saved as a preference pair: the prompt, the chosen answer and the rejected answer.

How does classic RLHF work?

Classic RLHF has two parts. First, the team trains a reward model: a separate model that reads a prompt and an answer, then gives a score. It learns from the preference pairs to give chosen answers higher scores than rejected ones.

Second comes reinforcement learning, which means learning by trying things and getting a reward. The model writes fresh answers, the reward model scores them, and the parameters get nudged toward higher scores. The 2022 study used a method called PPO, short for proximal policy optimization, for this step.

There is a catch. The model can learn to please the reward model without really getting better. This is called reward hacking. In a 2023 study, the reward gains from RLHF in its three test settings came largely from longer answers.

So RLHF adds a penalty that grows the further the model drifts from its instruction-tuned starting point. The 2022 study applied this penalty at every token, to stop the model over-optimising for the reward model.

What is DPO, and why is it simpler?

DPO skips the reward model and the trial-and-error loop. It reads each preference pair and directly pushes up the chosen answer's odds relative to the rejected one's.

It also compares against a reference model: a frozen, untouched copy of the starting model. Gains are measured relative to that copy, which keeps the tuned model close to where it started.

The 2023 paper that introduced DPO trains it with a simple classification loss, the kind used to sort things into two groups. On its tasks, it matched or beat the PPO-based approach, without sampling new answers during training and with little fiddling with training settings. For the helpline team, that means one training job instead of several models.

The diagram shows both routes from the same preference pairs. Notice that the DPO path has one box where RLHF has two.

Teams make a model safer by training it on examples of handling risky requests well, then attacking it on purpose to find gaps. This usually reuses the instruction and preference tuning methods, not one magic final step.

The helpline team runs safety as its own short round of both methods, so it can test it separately. It writes 3,000 safety examples. Some refuse clearly harmful requests, like using pesticide to hurt someone. Others help with safe questions that only sound risky, like how much pesticide to spray per acre.

A helpline model that refuses every pesticide question is useless to farmers. Refusing safe requests is called over-refusal. A 2023 study found that a few hundred safety examples made a model much safer, but too many made it refuse safe prompts that only looked unsafe.

Then comes red-teaming: a group tries hard to make the model misbehave on purpose, using tricks, role-play and odd wording. Failures go back into the safety examples and preference pairs.

The team red-teams in all seven languages. A 2023 study tested this on two widely used models. Prompts in low-resource languages, ones with little text online, were about three times as likely to get harmful replies as prompts in high-resource languages.

You evaluate an LLM after every stage with fixed test sets, human review and safety tests, then compare the new checkpoint with the last good one. A checkpoint moves forward only if it improved where expected and got no worse anywhere else.

The team fixes the tests and the pass mark before each stage starts. If you choose tests after seeing results, it is too easy to pick flattering ones. These are its checks:

• After pretraining: loss on held-out text, meaning text kept aside and never trained on, plus quiz-style tests in each language.
• After instruction tuning: whether it answers the actual question, in the caller's language and in the right format.
• After preference tuning: native speakers compare the new model's answers with the old one's, and the team tracks how often the new one wins.
• After safety tuning: the share of harmful prompts it refuses, and the share of safe prompts it wrongly refuses.

Comparing with the last good checkpoint catches regressions, meaning things that got worse. In the scenario, preference tuning improved Kannada answers but made Tamil arithmetic worse. The team traced it to Tamil pairs where raters had picked confident but wrong sums, removed them and ran the stage again.

Notice the question box near the end of the diagram. A No sends the team back to fix the data, never forward to the next stage.

Watch for contamination, where test questions leak into the training data. The model then scores well because it memorised answers, not because it learned. Validation examples are text kept aside for testing. The 2021 deduplication study found near-copies in training data for 4.6% of one widely used web-text dataset's validation examples, and 14.4% for a news dataset.

There is no single price: cost depends on model size, amount of text, test runs and paid human data. The helpline model's made-up total is about Rs 1 crore, and the largest models cost far more.

Notice in the table where the money goes. One lakh is 100,000 rupees.

Two patterns stand out. Pretraining is the single biggest bill, and most of it is GPU time. At a made-up Rs 200 per GPU per hour, the final week-long run on 32 GPUs costs about Rs 11 lakh. In the scenario, small test runs and restarts after crashes roughly triple that (made up).

The later stages are cheap in GPU time but expensive in people. Here, paying native-speaker writers and raters costs more than pretraining itself.

Training is only the first bill. Every later answer also costs money, which is called inference cost. For services with very heavy traffic, answering costs can rival training, so size choices should weigh both. At about 3,000 calls a day, the helpline is nowhere near that point.

Real pipelines catch exact copies with hashes and near-copies with similarity checks over overlapping word chunks. A hash is a short code computed from the text: identical texts always give the same hash.

The code below shows the exact-copy half. It tidies spaces first, so a page that differs only in spacing becomes an exact copy.

import hashlib

def clean(pages, min_words=5):
    seen, kept = set(), []
    for page in pages:
        text = " ".join(page.split())        # tidy spaces and line breaks
        if len(text.split()) < min_words:    # too short to teach much
            continue
        digest = hashlib.sha256(text.encode("utf-8")).hexdigest()
        if digest in seen:                   # exact copy seen before
            continue
        seen.add(digest)
        kept.append(text)
    return kept

pages = [
    "Sow ragi in Mandya after the first good rains.",
    "Sow ragi in  Mandya after the first good rains.",
    "Accept cookies",
    "Spray neem oil in the evening, not at noon.",
]
print(len(clean(pages)), "of", len(pages), "pages kept")

It prints 2 of 4 pages kept. The second page differed only by a double space, so after tidying it had the same hash as the first. Accept cookies was too short. Real pipelines add the language and quality filters here too.

Hashes miss the same article with a different header. The 2021 deduplication study handled those in two ways. One compared sets of five-word chunks between documents, using a fast estimate called MinHash, and flagged pairs over 80% similar.

The other found any repeated run of 50 or more tokens across the whole dataset, using a sorted index of every suffix. It then removed the duplicated text.

A common estimate is 6 floating-point operations per parameter per training token. A floating-point operation, or FLOP, is one multiply or add on decimal numbers. The 2022 compute study uses this rule in parts of its analysis, so the helpline run needs about 2.4 thousand billion billion FLOPs.

To turn that into time, divide by what each GPU really delivers. One common data-centre GPU, the A100, lists a peak of 312 trillion FLOPs per second for standard 16-bit maths. Real runs reach only part of that peak, so the code assumes 40%. That 40% is our assumption: measure your own.

params = 2e9            # 2 billion parameters
tokens = 200e9          # 200 billion training tokens
flops = 6 * params * tokens

peak = 312e12           # spec-sheet peak, 16-bit maths, one GPU
useful = 0.40 * peak    # assume 40% of peak is useful work
gpu_hours = flops / useful / 3600

print(f"{flops:.1e} FLOPs")
print(f"{gpu_hours:,.0f} GPU-hours")
print(f"{gpu_hours / 32 / 24:.1f} days on 32 GPUs")
print(f"{tokens / params:.0f} tokens per parameter")

It prints 2.4e+21 FLOPs, 5,342 GPU-hours, 7.0 days on 32 GPUs and 100 tokens per parameter. So the week in the scenario holds up, if the run keeps 40% of peak.

The last line matters too. 100 tokens per parameter is about five times the roughly 20 that the 2022 study's tables suggest. That is deliberate: extra training text makes a small model better without making it slower or dearer to serve. At the helpline's call volume, the 2023 study's inference-cost argument is weak.

The DPO loss looks at how much more the model now favours the chosen answer than the reference model did, minus the same for the rejected answer. It squashes that gap into a probability between 0 and 1 with the sigmoid function, an S-shaped curve. Then it takes the negative logarithm, like the pretraining loss.

Each input is a log-probability of a whole answer: the logarithm of the chance the model gives that answer, so always a negative number here. The setting beta controls how far the model may drift from the reference; the DPO paper mostly used 0.1.

import math

def dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1):
    # Each value is the log-probability of a whole answer.
    margin = (pol_chosen - ref_chosen) - (pol_rejected - ref_rejected)
    return -math.log(1 / (1 + math.exp(-beta * margin)))

print(round(dpo_loss(-40, -40, -40, -40), 3))   # start: model equals reference
print(round(dpo_loss(-35, -45, -40, -40), 3))   # chosen up 5, rejected down 5

It prints 0.693 and then 0.313. At the start, the model equals the reference, the gap is zero and the loss is the logarithm of 2. Raise the chosen answer by 5 and lower the rejected one by 5, and the loss drops.

What just happened? DPO rewarded the change in the gap, not the raw scores. A model that raised both answers equally would see no change in loss. That is how the reference copy keeps tuning anchored.

Explain it as a pipeline of stages, each with its own data, goal and test. Interviewers want to see that you know why each stage exists, not just its name.

1. 1

   Start with data

   Say where the text comes from and how you clean it: language, quality, copies, personal data and overlap with tests.

2. 2

   Size the pretraining run

   Next-token prediction over billions of tokens builds a base model. Estimate compute as 6 times parameters times tokens, and turn it into GPU-days.

3. 3

   Add instruction tuning

   Train on prompts with written answers, so the model learns to treat text as a request.

4. 4

   Add preference tuning

   Collect choices between two answers, then use RLHF or the simpler DPO, and say why you picked one.

5. 5

   Cover safety

   Train on refusals and on safe requests that sound risky. Then red-team in every language you support, and measure over-refusal.

6. 6

   Show the gates and the cost

   Test each stage against the last good checkpoint and watch for contamination. Give rough compute and people costs, and say they are estimates.