What is the difference between RAG and fine-tuning?

RAG looks up your documents at question time and puts the relevant parts in the prompt; the model stays the same. Fine-tuning trains the model on your examples and changes its weights. RAG is best for knowledge, fine-tuning for style, format or narrow repeated tasks.

When should I fine-tune an LLM?

After you have tried good prompting and RAG, and still need a consistent style, format or narrow skill across many requests. You also need a good set of examples, often hundreds or more, and tests that prove the tuned model is better.

Is RAG cheaper than fine-tuning?

Usually, both to start and to keep current. RAG needs a search store and adds tokens to each prompt, while fine-tuning has a training cost that repeats whenever things change. At very high volumes, a fine-tuned small model can sometimes be cheaper per request.

Can I use RAG and fine-tuning together?

Yes, and many mature apps do. A fine-tuned model can hold the house style while RAG supplies current facts, and the prompt sets the rules for each request.

Module 4 · Making AI know your stuff · Lesson 21 of 32

Prompting, RAG or fine-tuning?

10 minFree GenAI course · BeginnerLesson 4.4By Saket Bhatnagar

Researched and drafted with AI, reviewed and edited by Saket Bhatnagar.

In short

Prompting changes the instructions, RAG gives the model the facts, and fine-tuning changes the model’s habits.
Start with prompting. Add RAG when the model needs your knowledge. Fine-tune last, for a style or format prompting cannot hold.
Real apps often mix all three: a clear prompt, retrieved facts, and sometimes a tuned model.

Soon

Video lesson · coming soon

Prompting, RAG or fine-tuning? How to choose

Lala, your AI mentor, will walk you through this lesson in a short animated video. Until then, everything is here to read and try.

01 Three months of fine-tuning, wrong fees
02 Brief, recipe book, or cooking school
03 Words, facts, or habits?
04 Climb the ladder, stop early
05 Pick the tool
06 Next: AI that takes action

Arjun is building a helper for a coaching institute in Kota. In the first meeting, someone says, “Let’s fine-tune our own model on all our notes!” It sounds serious and impressive. Three months and a big bill later, the helper still gets this year’s fee structure wrong.

The idea was not silly. It was the wrong tool for the job. This lesson gives you a plain way to pick the right one, so you don’t spend three months finding out.

Three ways to get better answers

Think of it like this

Brief the cook, hand over a recipe book, or send them to cooking school

Meera hires a talented cook for her daughter’s wedding in Lucknow. First, she gives a brief: “Less oil, mild spice, serve at eight.” That is quick and free. Then she hands over Nani’s handwritten recipe book, so the family dishes come out exactly right. She could also send the cook on a months-long course in Nani’s style. That would change how the cook cooks, but it costs time and money, and the cook still won’t know next week’s guest list.

In real lifeIn AI

The quick brief: In AI: Prompting
Nani’s recipe book, open on the counter: In AI: RAG: facts looked up when needed
A months-long cooking course: In AI: Fine-tuning: changing the model itself
Next week’s guest list: In AI: Fresh facts no training can know in advance

You met prompting in module 3, and RAG earlier in this module. Fine-tuning is new, but you saw its cousin in the training lesson: the stage where a raw model practises being a helpful assistant. Fine-tuning is that same kind of practice, on your examples.

What does each one actually change?

The cleanest way to tell them apart is one question: what am I changing? The words going in, the facts going in, or the model itself?

Prompting, RAG and fine-tuning side by side
Question	Prompting	RAG	Fine-tuning
What changes?	The instructions you send	The facts you send	The model’s weights
Best for	Tone, format, task steps	Private, fresh or large knowledge	A fixed style, format or narrow skill, at scale
Effort to start	Minutes to hours	A few days for a simple version	Days to weeks, plus hundreds or more good examples
When facts change	Edit the prompt	Edit the documents	Train again
Can it show sources?	No	Yes, it can cite chunks	No
Main risk	Long prompts get costly	Bad search gives wrong answers	Cost, lost general skills, still makes things up

Now, the big myth to drop. Fine-tuning is a weak way to teach facts. A model picks up patterns well, but specific facts get blurred, mixed up or lost, and you cannot see where an answer came from. Research that compares the two has generally found RAG more reliable for adding knowledge.

That is what happened in Kota. When fees changed in April, the fine-tuned model kept quoting the old fees. It would keep doing so until someone trained it again.

Climb the ladder: prompt, then RAG, then fine-tune

Most teams climb a ladder, and stop at the lowest rung that works. Each rung up costs more, so you only climb when the tests say you must.

Which tool should you reach for?

Drag to explore

Start with a better prompt. If answers lack your facts, add RAG. Fine-tune only when a style or format still won’t stick after good prompting.

Try prompting first
Rewrite the brief: role, context, an example, the format. Many “we need fine-tuning” problems disappear here, in an afternoon.
Add RAG when knowledge is the gap
If the answer depends on your documents, or on facts that change, retrieve them. Now you can fix an answer by editing a file.
Fine-tune when habits are the gap
If you need the same special tone or format across thousands of requests, and the long prompt is getting costly, fine-tuning can build it in.
Measure at every rung
Keep a set of test questions with good answers. Move up only when the tests show the lower rung is not enough.

Mixing them in real apps

These three are not rivals. A grown-up AI app often uses all of them. The prompt sets the role and the rules. RAG brings in today’s facts. A fine-tuned model may keep a house style without a long prompt every time.

Back to Arjun in Kota. The real fix was not a bigger training run. It was a clear prompt (“You help students of this institute; answer kindly, in short points”), plus RAG over the current fee sheet, batch timings and refund rules. When fees changed, the office updated one document. The new helper was ready in about a week.

Try it yourselfPrompting, RAG or fine-tuning?Read each situation and pick the tool you would try first. The reason appears after you choose.

Check yourself

The Kota institute has hundreds of pages of fees, batch timings and rules, and they change every April. Which approach keeps answers right with the least ongoing work?

Pick an answer to see why it is right or wrong.

Check yourself

Which statement about fine-tuning is true?

Pick an answer to see why it is right or wrong.

Remember these three

Prompting changes instructions, RAG supplies facts, fine-tuning changes habits.
Climb the ladder: prompt first, RAG for knowledge, fine-tune last, and test at each rung.
For facts that change, reach for RAG, not fine-tuning.

Questions people ask

What is the difference between RAG and fine-tuning?: RAG looks up your documents at question time and puts the relevant parts in the prompt; the model stays the same. Fine-tuning trains the model on your examples and changes its weights. RAG is best for knowledge, fine-tuning for style, format or narrow repeated tasks.
When should I fine-tune an LLM?: After you have tried good prompting and RAG, and still need a consistent style, format or narrow skill across many requests. You also need a good set of examples, often hundreds or more, and tests that prove the tuned model is better.
Is RAG cheaper than fine-tuning?: Usually, both to start and to keep current. RAG needs a search store and adds tokens to each prompt, while fine-tuning has a training cost that repeats whenever things change. At very high volumes, a fine-tuned small model can sometimes be cheaper per request.
Can I use RAG and fine-tuning together?: Yes, and many mature apps do. A fine-tuned model can hold the house style while RAG supplies current facts, and the prompt sets the rules for each request.