Prompting vs RAG vs Fine-Tuning, Explained From Zero

When you want an AI to do something specific — answer from your data, adopt a style, handle your domain — there are three main ways to get there, and people constantly reach for the wrong one and waste weeks. This guide starts from nothing and builds up a decision framework, then shows how your evaluations make the choice on evidence rather than gut feel.

1 · Three ways to shape a model

The first is prompting: just write better instructions. The second is RAG, retrieval-augmented generation: give the model your documents to read at answer-time. The third is fine-tuning: actually retrain the model a little on your own examples. People constantly reach for the wrong one — usually the most complex — and waste time. The headline, which we'll unpack, is that these aren't equals to pick between freely; they form a ladder, and you should almost always climb it from the bottom, only moving up when the simpler rung genuinely isn't enough.

2 · Prompting: change the instructions

The first rung is prompting, and it's astonishing how far it gets you. You change nothing about the model — you just write clearer, better instructions: the role to play, the context, the format, a few examples of what good looks like. That's it. The advantages are huge: it's essentially free, you can change it in seconds, and you iterate as fast as you type. Everything from the prompt-engineering topics lives here — specificity, roles, few-shot examples, structured output. A remarkable share of “we need to customise the model” problems turn out to be “we need a better prompt.” So prompting is always where you start, because if it solves your problem — and it often does — you've saved all the cost and complexity of the other two. Only when you've genuinely pushed prompting to its limits and it still falls short should you climb higher.

3 · RAG: give it the knowledge

The second rung is RAG, and the key to the whole decision is knowing which problem it solves. RAG is the answer when the issue is knowledge — the model doesn't know something because it's private, specific, or newer than its training. You retrieve your relevant documents and hand them to the model to read. So the diagnostic question is: “Is my model failing because it lacks information, or because it lacks a skill or style?” If it's information — “answer questions about our product,” “use our internal policies,” “reference current data” — RAG is almost always right, and it's the default for anything document-based. It's more work than prompting but far less than fine-tuning, and it has a decisive bonus: update the documents and the knowledge updates instantly, no retraining. Most business AI needs are knowledge needs, which is why RAG is so often the destination.

4 · Fine-tuning: retrain the behaviour

The third rung is fine-tuning, the most powerful and most misunderstood. Here you actually retrain the model a bit, showing it many examples of the behaviour you want until that behaviour becomes its new default. The crucial point — the one people get wrong most — is what fine-tuning is for: teaching a skill or style, not adding knowledge. Good uses: making the model reliably output a very specific format, adopt a consistent brand voice, or handle a specialised task in a particular way — behaviours hard to fully capture in a prompt. Bad use: trying to cram facts in. If you want the model to know your documents, RAG does that better, more cheaply, and stays current — facts fine-tuned in are frozen and go stale. Fine-tuning also costs the most: lots of quality examples, real expertise, and redoing it as things change. It's the specialist tool for shaping how the model behaves, not what it knows.

5 · Knowledge or behaviour?

Here's the single question that resolves most of these decisions, because it cuts through the confusion instantly. When your model isn't doing what you want, ask: is it missing knowledge, or missing a behaviour? If it's missing knowledge — a fact, a document, your specific data — the answer is RAG: give it the information to read. If it's missing a behaviour — a format, a style, a specialised skill — then first try to prompt that behaviour, and only fine-tune if prompting can't get you there. Notice that “knowledge” almost never points to fine-tuning, and “behaviour” usually points to prompting first. People burn weeks fine-tuning a model to “know” their documents when RAG would have done it in an afternoon and stayed current. Keep this question in your pocket — knowledge or behaviour? — and you'll route yourself to the right tool almost every time.

6 · Climb the ladder in order

Here's the strategy that ties it together: treat these three as a ladder and climb from the bottom. Always start with prompting — fastest and cheapest, and it solves more than you'd expect. If prompting genuinely can't do it and the gap is knowledge, add RAG. Only if you've truly exhausted prompting and RAG, and you specifically need a behaviour that neither can deliver, do you reach for fine-tuning. The reason for the order is simple: each rung up costs dramatically more in effort, money, and ongoing maintenance, so you want to stop climbing the instant your problem is solved. The classic, expensive mistake is jumping straight to fine-tuning because it sounds the most serious — when a better prompt or a bit of RAG would have solved it in a fraction of the time. And these combine happily: a great system often uses a strong prompt and RAG together, with fine-tuning reserved for the rare case that truly needs it.

7 · Let the eval decide

Here's where the last article pays off directly. This decision — is prompting enough? does RAG help? is fine-tuning worth it? — is not one to settle by opinion or gut. It's exactly what evaluations are for. Build your test set, then measure each approach against it: score the prompt-only version, score it with RAG added, and if tempted by fine-tuning, score that too. Now the ladder isn't a debate — the numbers tell you when a rung is good enough and when climbing higher genuinely buys something. Maybe RAG lifts your score from 70 to 92 and fine-tuning only nudges it to 93 for ten times the cost — the eval makes that trade-off obvious. Without evals you'd argue in circles; with them you climb on evidence, stopping at the rung that clears your quality bar most cheaply. Measure, then decide — that's how this choice should always be made.

8 · A simple test you can run this week

9 · Glossary — every term, spelled out

Prompting

Changing only the instructions you give the model — the fastest, cheapest approach, tried first.

RAG

Giving the model your documents to read at answer-time — the fix for missing knowledge.

Fine-tuning

Retraining the model on examples so a behaviour becomes its default — for skill and style, not facts.

Knowledge vs behaviour

The routing question: missing information points to RAG; a missing skill or style points to prompting then fine-tuning.

The ladder

Trying prompting, then RAG, then fine-tuning in order, stopping as soon as the problem is solved.

Eval-driven choice

Deciding between approaches by measuring each on a test set rather than by gut feel.