RAG (Retrieval-Augmented Generation), Explained From Zero

The single most common thing people want from AI at work is simple to state: “answer questions using our documents.” A model on its own can't do that reliably — but a pattern called RAG can, and it's the workhorse behind most useful business AI. This guide starts from nothing and builds up exactly what RAG is, why it works, and where it still needs care.

1 · The closed-book problem

A language model learned from a huge pile of text during training, and then that training stopped. So its knowledge is like a student sitting a closed-book exam: whatever they memorised, they can use; everything else, they must guess. That creates three hard limits. It has never seen your private material — your internal wiki, your contracts, last week's meeting notes — because none of it was in its training. It's frozen at a past date, so anything newer is unknown. And when it doesn't know, it tends to produce a confident, plausible-sounding guess anyway — a hallucination. Asking such a model about your specific documents is asking a student to recite from a book they were never given. RAG hands them the book.

2 · The open-book idea

RAG's insight is almost obvious once said: don't rely on what the model memorised — give it the relevant material to read at the moment it answers. Before it responds, you find the passages from your documents that bear on the question and place them right in front of it, then ask it to answer using those passages. Now it's an open-book exam: the model still supplies the language and reasoning, but the facts come from the text you provided. The name spells out the three steps — retrieval (find the passages), augmented (add them to the prompt), generation (the model writes the answer). Retrieve, augment, generate. That's the entire pattern, and everything else is detail.

3 · Retrieval: finding the right passages

The first step is finding the handful of passages actually relevant to the question — and this is where the previous topic pays off. Your documents are split into pieces and embedded, turning each into a point on a map of meaning; the question is embedded too, and vector search returns its nearest neighbours — the pieces closest in meaning. So a question about “time off” surfaces the policy section on “annual leave” even with no shared words, because retrieval works by meaning, not keywords. This step is the make-or-break of RAG: if retrieval fetches the right passage, the model has what it needs; if it fetches the wrong one, no amount of model brilliance can save the answer. Good retrieval is the foundation the whole pattern stands on.

4 · Augment: handing them to the model

Once you have the relevant passages, you augment the prompt — a plain word for “add them in.” You build a combined prompt that says, in effect: “Here are some passages from our documents. Using only these, answer the user's question,” followed by the retrieved text and then the question. That instruction matters as much as the passages: it tells the model to lean on the provided text rather than its own memory, and to say when the passages don't contain the answer. This is the “augmented” in retrieval-augmented generation — the model's built-in knowledge, augmented at answer-time with fresh, specific, trustworthy material it can read directly. You're not retraining anything; you're just handing over the right book, opened to the right page.

5 · Generate: an answer grounded in the text

Finally the model generates the answer — but now it's reading from the passages you gave it rather than dredging its memory. Because the source text is right there, the answer is grounded: it reflects what your documents actually say. And since you know which passages went in, the model can point back to them — “according to the leave policy, you get 20 days” — giving you citations you can verify. This is the finished loop: a question comes in, retrieval finds the relevant pieces, they're added to the prompt, and the model produces a natural-language answer anchored in your real content. The user just asks a question and gets a trustworthy answer; the retrieve-augment-generate machinery runs quietly underneath.

6 · Three wins: private, current, citable

RAG directly cures the three closed-book weaknesses. Private knowledge: the model can now answer from documents it was never trained on — your wiki, your policies, your data — simply because you retrieve and hand them over at answer-time. Currency: update the documents and the answers update, no retraining needed; retrieve today's version and the model reads today's facts. Citations: because the answer is built from specific passages, you can show where each claim came from, which is exactly what people need to trust and check an AI's output. Together these turn a model from an unreliable know-it-all into a grounded assistant for your world — which is why RAG, not fine-tuning, is the default first move for “answer from our documents.”

7 · Where RAG can still go wrong

RAG is powerful but not automatic, and its failures almost always trace to retrieval. If the right passage isn't fetched, the model answers from a gap — and may hallucinate to fill it — so quality lives or dies on getting the relevant text in front of it. That, in turn, depends on how you split documents into pieces: chunks that are too big, too small, or cut mid-idea wreck retrieval, which is why chunking gets its own topic next. Two more habits matter: instruct the model to say “the documents don't cover this” rather than guess, and enforce that retrieval only returns what the user is allowed to see — never let RAG surface a document a person shouldn't read. When a RAG system disappoints, look first at what it retrieved: read the passages it pulled, and the problem is usually right there.

8 · A simple test you can run this week

9 · Glossary — every term, spelled out

RAG (retrieval-augmented generation)

Fetching relevant passages from your documents and giving them to the model to answer from, rather than relying on its memory.

Retrieval

Finding the passages most relevant to a question, usually via vector search over your documents.

Augment

Adding the retrieved passages into the prompt so the model can read them at answer-time.

Grounded

An answer based on provided source text rather than the model's own recollection.

Citation

A pointer back to the specific passage a claim came from, so it can be verified.

Closed-book vs open-book

Answering from memory alone versus answering while reading the relevant material — RAG makes the model open-book.