Fine-Tuning vs RAG vs Prompting: Which One Do You Need?

You pay weeks of work for a model that still quotes old facts with full confidence. Prompting, RAG and fine-tuning each fix a different problem, so fine-tuning vs RAG is really a question about which problem you have.

Picture a common setup. An event-planning business in Lucknow sells 14 packages, from a Rs 1.2 lakh (120,000) sangeet evening to a Rs 9 lakh three-day wedding. About 600 enquiries arrive each month on WhatsApp, many late at night. The setup is made up, but the problem is real.

The owner wants an assistant that answers those enquiries. It must know the packages and sound like the business: polite, warm and short. The business has one engineer and two months.

A software seller has already offered to train a custom model on the brochures for Rs 4 lakh. The owner likes the idea of a model that is truly his own. The engineer has to say whether it is worth the money.

The engineer sees two needs hiding inside one wish. The assistant needs facts: what each package includes, what it costs and which dates are free. It also needs a manner: how a reply opens, how long it runs and how it ends.

• Facts change. Prices go up each wedding season, and venue dates fill up every week.
• Manner barely changes. The owner has wanted the same polite tone for twenty years.
• Facts must be exactly right. A wrong price quoted to a family causes a real argument later.
• Manner can be slightly off without harm, but it shapes how customers feel about the business.

Keep those two needs apart. The rest of this guide shows that facts and manner need different fixes, and that the seller's offer aims the most expensive fix at the wrong need.

Prompting changes what you tell the model, RAG changes what the model can read for each question, and fine-tuning changes the model itself. The first two leave the model untouched.

Model. The model is the program that reads text and writes text back. Its knowledge and habits live in its weights: billions of numbers set during training. The business rents a general model from a model service, which charges for each use.

Prompting. A prompt is all the text your program sends to the model with a request. Prompting means improving that text with clearer rules and a few worked examples. Example: start every reply with Aadab, a polite Lucknow greeting, and stay under 80 words.

RAG. RAG, short for retrieval-augmented generation, means searching your own documents for each question and pasting the best matches into the prompt. Example: a question about mehendi nights pulls the current Mehendi Night package details into the prompt.

Fine-tuning. Fine-tuning means training an existing model a little more on your own examples. It changes the model's weights, or adds a small set of new ones beside them. Example: showing the model 500 of the business's best past replies, so it picks up the house style.

The 2020 paper that introduced RAG gives a useful pair of names. Knowledge stored in the weights is parametric memory. Knowledge kept outside the model, in a searchable index, is non-parametric memory, and RAG adds the second kind to the first.

Picture a family hiring a pandit, a priest, for a griha pravesh, the puja for a new home. Three things shape how well he does the job.

• Prompting is the slip of paper the family hands him: their names, their gotra (family lineage) and the rooms to bless.
• RAG is today's panchang, the almanac with date-specific facts, such as the muhurat, the auspicious time to begin.
• Fine-tuning is the years of practice that shaped how he performs every puja, without being reminded.

Where the picture breaks: a pandit understands what he reads, while a model can misread a retrieved document or ignore it. And a pandit remembers this family next year, while a model forgets each conversation unless you send it again.

The diagram shows where each approach acts when a customer message comes in. Follow the arrows, and notice that fine-tuning touches only one box: the model.

Yes, in almost every case. Prompting is the cheapest change to try, it takes minutes instead of weeks, and it shows you which problems are left for RAG or fine-tuning.

The engineer's first version is only a prompt. It holds the owner's tone rules, three example replies he picked and the full price list pasted at the bottom. Putting a few examples inside the prompt is called few-shot prompting.

Before trying it, the engineer builds a test set: 80 old enquiries with known good answers. The first prompt gets the tone right in about 7 of 10 replies and the facts right in about 8 of 10 (made up). That is a fair start for one afternoon. Then the problems show up, pointing in two directions.

• The price list is 40 pages, about 30,000 tokens. A token is a small piece of text, often a word or part of one, and services charge per token.
• Prices change every season. Each change means editing the prompt by hand, and one missed edit means a wrong quote.
• Venue dates change daily, and no hand-edited prompt can keep up with that.
• In long chats the tone slips. By the tenth message, replies grow longer and drop the greeting.

The first three are knowledge problems: the model lacks current facts. The last one is a behaviour problem: the model has the facts but writes them the wrong way.

Many model services can also cache a long, repeated start of a prompt, which cuts its cost. Our guide on prompt caching explains how. Caching makes a big prompt cheaper, but it does nothing to keep prices fresh.

For company knowledge, RAG is usually better than fine-tuning. RAG reads your current documents when it answers, while fine-tuning bakes facts into weights that go stale and are hard to check.

Research backs this up. A 2023 study kept training models on plain text about a topic, then compared that with RAG on question tests. RAG did better consistently, both for facts the models had seen before and for brand-new ones.

A 2024 study found something worse. In its question-answering tests, the model learned new facts from fine-tuning much more slowly than facts it already knew. As it finally learned them, it became more likely to hallucinate, meaning to state false things confidently.

That settles the seller's pitch. Training on brochures would bake in this season's prices, and by March they would be wrong. The model would quote them with full confidence, and the business would have to train again every season.

RAG fits the facts far better. The engineer turns each package into a short package card in a database. The sales team edits a card when a price changes, and the next customer sees the new price without any training.

Here is the prompt builder, small enough to run. The word-matching search is a stand-in for real search, and the three cards stand in for the 14 real ones.

import re

CARDS = [
    {"name": "Sangeet Evening", "includes": "DJ, stage and dinner for 150 guests", "price": 120000},
    {"name": "Mehendi Night", "includes": "artists, decor and snacks for 150 guests", "price": 180000},
    {"name": "Three-Day Wedding", "includes": "venue, decor and catering for 400 guests", "price": 900000},
]

TONE_RULES = "Start with 'Aadab'. Use 'aap'. Stay under 80 words. End with one next step."

def search_package_cards(message, top_k=2):
    # Stand-in for real search: count words shared with each card name
    words = set(re.findall(r"[a-z]+", message.lower()))
    scored = [(len(words & set(c["name"].lower().split())), c) for c in CARDS]
    scored.sort(key=lambda pair: pair[0], reverse=True)
    return [c for score, c in scored[:top_k] if score > 0]

def build_prompt(message):
    cards = search_package_cards(message)  # RAG: fetched fresh for every message
    facts = "\n".join(f"{c['name']}: {c['includes']}. From Rs {c['price']:,}" for c in cards)
    return (f"{TONE_RULES}\n\nUse only these packages and prices:\n{facts}\n\n"
            f"Customer: {message}")

print(build_prompt("Mehendi night for 150 guests in December?"))

Run it with Python 3 and it prints the four tone rules, then one line for the Mehendi Night card ending From Rs 180,000, then the customer's message. The sangeet and wedding cards share no words with the question, so they stay out.

What just happened? The search ran for this one message, so a price edited this morning is used this afternoon. And the line that says use only these packages and prices tells the model not to quote from memory. RAG brings three more benefits.

• Checkable: each reply can name the package card it used, so a planner can verify it in seconds.
• Removable: delete a wrong card and the fact is gone. Trained weights have no simple delete for one fact.
• Controllable: you can keep internal documents, such as supplier rates, away from customers. Our guide on RAG access control covers this.

RAG has its own costs. You must split documents into pieces, search them well and keep the index current. The engineer's first search sometimes pulled the sangeet card for a mehendi question, and better search is covered in our guide on naive, advanced and modular RAG.

Fine-tuning is right when the model must behave a certain way every time: a fixed format, a house style, a narrow task, or a smaller and cheaper model. It is the wrong choice for facts that change.

Back to the pandit. Years of practice are worth it for a habit you want at every puja, without a reminder. Fine-tuning works the same way: it moves instructions you keep repeating into the model itself.

• Format: every reply must follow a fixed shape, such as a three-line summary of package name, inclusions and starting price.
• Voice: a tone that a few examples in the prompt cannot hold through long chats.
• A narrow task at high volume: sorting enquiries into wedding, corporate or birthday, thousands of times a day.
• A smaller model: a small model trained for one narrow task can sometimes match a much larger general model on that task, with cheaper and faster replies.

Research supports the format point. A 2023 study fine-tuned a 65-billion-parameter model (parameters is another word for weights) on just 1,000 carefully chosen examples, and it learned to follow specific reply formats. The authors argue that almost all knowledge comes from pretraining, the first and biggest round of training.

The small-model point has evidence too, with a catch. A 2023 study trained a 770-million-parameter model on a task, using a larger model's worked reasoning as extra training data. On one benchmark, a standard public test set, it beat a 540-billion-parameter model prompted with a few examples, using 80 percent of the data.

The catch is in the same paper. Plain fine-tuning of that small model, without the extra reasoning data, struggled to match the large model even with all the data. So a small tuned model can win, but only on a narrow task, and how you train it matters.

Fine-tuning often does not retrain the whole model. Many teams use LoRA, short for low-rank adaptation, which freezes the original weights and trains two small extra tables of numbers beside chosen parts of the model.

The LoRA paper compared this with full fine-tuning of a 175-billion-parameter model using Adam, the usual training method. It reports 10,000 times fewer trainable weights and a third of the GPU memory, the memory on the chips that run models.

The paper also reports no extra delay per answer, because the small tables can be added into the original weights before serving. Our guide on LoRA and QLoRA shows that maths step by step.

What data does fine-tuning need?

Fine-tuning needs examples of exactly the behaviour you want: an input and the ideal output, written or approved by people who know the job. Quality beats quantity here.

The 1,000-example study tested this. In its extra experiments, growing quality-filtered training data from 2,000 to 32,000 examples did not improve answers. Filtered data scored about half a point higher than unfiltered data on a 1 to 6 helpfulness scale, graded by another AI model.

The engineer builds the training set from old WhatsApp chats. The owner rates 900 of them, and the engineer keeps the 600 he scores 4 or 5 out of 5. Customer names and phone numbers are removed before anything leaves the business.

One choice in this data matters most. Each example includes the package cards that were shown that day, not only the reply. So the model learns to copy prices from the cards in front of it, instead of remembering them.

JSON is a common text format for data. Many fine-tuning services accept one JSON object per line, a format called JSONL, and a chat shape like the one below is common. Check your own service's documentation for its exact field names.

import json

approved_chats = [{
    "cards": "Mehendi Night: artists, decor and snacks for 150 guests. From Rs 180,000",
    "message": "Mehendi ke liye kitna lagega?",
    "reply": "Aadab! Mehendi Night aapke liye Rs 1,80,000 se shuru hota hai. Date bata dijiye?",
}]

def to_training_example(chat):
    return {"messages": [
        {"role": "user", "content": f"Packages:\n{chat['cards']}\n\nCustomer: {chat['message']}"},
        {"role": "assistant", "content": chat["reply"]},
    ]}

with open("voice.jsonl", "w", encoding="utf-8") as f:
    for chat in approved_chats:
        f.write(json.dumps(to_training_example(chat), ensure_ascii=False) + "\n")

line = open("voice.jsonl", encoding="utf-8").readline()
print(len(json.loads(line)["messages"]), "messages in the first line")
print(json.dumps("आदाब"), json.dumps("आदाब", ensure_ascii=False))

Run it and it prints 2 messages in the first line, then the greeting twice. The first copy is escape codes starting with a backslash and u0906, and the second is the readable word in Devanagari, the script Hindi uses.

What just happened? The user message puts the package cards above the customer's message, exactly as the live assistant will. Setting ensure_ascii to False keeps Hindi or Urdu text readable in the file, which makes reviewing examples by eye far easier.

Prompting costs the least to start, RAG adds steady building and running costs, and fine-tuning adds the largest upfront and upkeep costs. The real bill depends on your traffic, your data and how often things change.

Look at cost in four buckets: building it, running each reply, keeping it fresh and testing every change. The table compares the three approaches for the event business, with made-up rupee figures.

Two rows surprise people. RAG can cost less per reply than prompting, because it sends a few package cards instead of 40 pages. A small tuned model reading the same cards can be cheaper still, because a small model usually costs less per token and needs no long instructions.

Check the per-reply figures yourself. At a made-up rate of Rs 0.07 per 1,000 input tokens, 30,000 tokens cost Rs 2.10 and a 5,000-token RAG prompt costs Rs 0.35.

Now the monthly sums. Six hundred enquiries with about 5 replies each make about 3,000 replies. Prompting with the full list costs about Rs 6,300 a month, and RAG brings that to about Rs 1,050.

The hidden cost of fine-tuning is upkeep. A tuned model is tied to the base model it started from, and a LoRA add-on works only on top of that exact model. If your service retires that base model, or a better one arrives, you train again.

You test all three the same way: run a fixed set of real questions with known good answers before and after each change, and score facts and behaviour separately. Without that test set, every choice is a guess.

The engineer's test set has the 80 past enquiries. Each lists the right packages and prices, plus a short tone checklist from the owner. Every reply gets scored in three ways.

• Fact accuracy, checked by code. Every price in a reply must match a price on the package cards.
• Voice and format, checked against the list: opens with Aadab, uses the respectful aap, stays under 80 words, ends with one next step.
• A human read of 20 replies each round, because checklists miss replies that simply feel wrong.

Pause and predict. The engineer adds RAG first, then a fine-tuned small model on top. Which score will each change move, and which will it leave alone?

Here are the results (made up). Prompting alone got facts right 81 percent of the time and voice right 72 percent. Adding RAG lifted facts to 97 percent, while voice moved only to 74 percent.

RAG with a fine-tuned model kept facts at 97 percent and lifted voice to 93 percent. RAG moved facts and barely touched voice, and fine-tuning did the opposite.

Decide by asking what is actually wrong with the replies, in a fixed order. Try prompting first, add RAG for missing or changing facts, and fine-tune only for behaviour that prompting cannot hold.

The flowchart shows that order. Start at the top box, and notice that every fix sends you back to the test set. A few rules of thumb then cover most real cases.

• If a fact would change before you could train again, it belongs in RAG.
• If you keep repeating the same instruction and the model keeps slipping, consider fine-tuning.
• If you cannot collect a few hundred good examples, you are not ready to fine-tune.
• If you have no test set yet, build that before any of the three.

The business never paid the seller's Rs 4 lakh. RAG plus a small tuned model cost about Rs 2.1 lakh upfront (made up), about half, and its prices stay current.

In one controlled 2024 study, the extra hallucination came from the training examples the model did not already know. The model learned those late, and fitting them made it more likely to state wrong answers confidently.

The authors fine-tuned a model on closed-book questions, where the model answers from memory with no documents. They sorted each training example by whether the model already knew the answer before fine-tuning.

• Examples with unknown answers were learned substantially slower than known ones, mostly late in training.
• Stopping training at the best score on a check set left the unknown examples roughly harmless. Training on until they were fitted made them very harmful.
• Removing the unknown examples from the training set was a second way to avoid the harm, which the authors call initial evidence.
• The study covered only short question answering; the authors say longer answers still need checking.

The 2023 knowledge study adds the other half. Its fine-tuning was plain continued training on raw text cut into 256-token chunks. Accuracy on new facts rose as each chunk was paraphrased more times, up to the ten paraphrases it tested.

On its new-facts test, combining that fine-tuned model with RAG did not beat RAG alone. The authors call their repetition idea a hypothesis that needs more research.

For the event business, the lesson is about the voice data. If a training reply quotes a price that is not in that example's input, the model must memorise it, which is new knowledge. Putting the day's cards in every input turns that into copying instead. That is our reading of the two studies, not something either one tested.

Yes. RAG supplies the current facts, and a fine-tuned model turns them into replies with the right format and voice. The hard part is teaching the tuned model to ignore a wrong card when search returns one.

A 2024 method called RAFT, short for retrieval-augmented fine-tuning, trains for exactly that. In the paper's main setup, each training question comes with the document that answers it plus four distractors, documents that look related but do not help.

For some questions, RAFT leaves out the right document but still trains on the correct answer. The authors say this pushes the model to memorise answers. That suits a fixed subject, but not prices that change.

Training answers quote the helpful passage and reason step by step. The paper found that training with only the right document often did worse than mixing in distractors. The best share of examples that keep the right document varied by dataset, from 40 to 100 percent, so treat it as something to tune.

For the event business, borrow the distractor half and skip the memorising half. Some training examples show the sangeet card beside the right card on a mehendi question, with a reply that ignores it, and every price question keeps its right card.

The diagram shows the system the engineer ships. Follow a message from the customer, and notice where prices get checked.

One serving detail matters with LoRA. The paper's no-extra-delay claim holds when the small tables are added into the weights. The paper notes that one merged model then cannot easily serve requests for different tasks in a single batch, a group of requests run together.

The paper suggests keeping add-ons unmerged and picking one per request only where some extra delay is acceptable. One business with one voice can simply merge.

Check every price in the reply against the cards in code, before sending. Even with RAG and a tuned model, a model can misread a card, so the last check should not be the model itself.

Rupee prices are tricky to match. The same amount can appear as Rs 1,80,000 in Indian digit grouping, Rs 180,000 in Western grouping, or Rs 1.8 lakh. The guard below turns all three into one number before comparing.

import re

def prices_in(text):
    """Find rupee amounts such as Rs 1,80,000 or Rs 180,000 or Rs 1.8 lakh."""
    found = []
    for num, lakh in re.findall(r"(?:Rs\.?|₹)\s*(\d[\d,]*(?:\.\d+)?)\s*(lakh)?", text):
        value = float(num.replace(",", ""))
        found.append(round(value * 100000) if lakh else round(value))
    return found

def price_check(reply, cards):
    allowed = {c["price"] for c in cards}
    wrong = [p for p in prices_in(reply) if p not in allowed]
    return "send" if not wrong else f"to planner, unknown price {wrong}"

cards = [{"name": "Mehendi Night", "price": 180000}]
print(price_check("Aadab! Mehendi Night is Rs 1,80,000.", cards))
print(price_check("Aadab! Mehendi Night is Rs 1.8 lakh.", cards))
print(price_check("Aadab! Mehendi Night is ₹1,65,000.", cards))

Run it and it prints send, send, then to planner, unknown price [165000]. The first two replies give the card's price in two formats, and the third gives an old price.

What just happened? The guard ignores digit grouping and compares numbers in rupees. It only finds amounts written in digits after Rs or ₹. A price written as INR 1,65,000, as 1,65,000 rupees or in words passes unchecked, so add such replies to your test set.

Explain it as a diagnosis, not a contest between tools. Split the problem into knowledge and behaviour, then justify each fix with cost, freshness and a test.

1. 1

   Split the requirement

   Name the knowledge the system needs and the behaviour it needs. For the event business, that means package facts and a short, polite house style.

2. 2

   Start with prompting and a test set

   Say you would begin with instructions and a few examples, and measure facts and voice separately on real past questions.

3. 3

   Put changing facts in RAG

   Prices and dates change weekly, so they live in a searchable store that the sales team updates without retraining.

4. 4

   Fine-tune only for behaviour

   Use LoRA on a small model for format and voice, with a few hundred approved examples whose inputs contain every fact the reply uses.

5. 5

   Cover cost and upkeep

   Work out cost per reply out loud, and mention retraining when the base model is retired or replaced.

6. 6

   Close with guards

   Describe the price check in code, distractor cards in training and the handover to a human planner.