Key insight

Before you can search documents by meaning, you must split them into chunks — pieces small enough that each is about one idea — then embed and store them in an index. Chunk size is a Goldilocks problem: too big blurs the meaning, too small loses context. Get it right and RAG just works; get it wrong and even a great model gives bad answers, because most RAG failures are chunking failures in disguise.

Chunking is the least glamorous part of building an AI app and, quietly, one of the most important. It's the prep work that decides whether retrieval hands the model a clear, relevant passage or a blurry, useless one — and therefore whether the whole thing works. This guide starts from nothing and builds up the handful of judgements that separate a RAG system that just works from one that mysteriously doesn't.

1 · Why not embed the whole document

You might expect to embed a whole document as one point on the map of meaning — but that fails, and understanding why explains everything that follows. A long document is about many things, so its single embedding is a blurry average of all of them: a point that's vaguely near everything and truly close to nothing. Embed a whole employee handbook and its one dot sits in a vague middle, not sharply near “vacation days” or “expenses” or “parking” — so a question about vacation days never lands cleanly on it. The fix is to split the handbook into its sections first: each becomes a sharp point about one clear idea, and the vacation chunk sits exactly where vacation questions land. Embeddings capture meaning best when the text is about one thing — so you chunk documents down toward one idea each.

A whole document blurs; a chunk is sharp A whole document embeds as one vague point; a single chunk embeds as one clear point. whole doc = one blurry dotmeaning averaged into mush one chunk = one sharp dota single clear idea chunk down toward one idea per piece
Figure 1. The reason chunking exists: a point can only sharply represent one meaning, so each chunk should hold roughly one idea.

2 · The Goldilocks problem of chunk size

Chunk size turns out to be a Goldilocks problem — there's a “just right,” and both extremes hurt. Too big and you're back to the blurry average: the chunk mixes several ideas, its point is fuzzy, and retrieval gets vague matches. Too small — a lone sentence like “it must be renewed annually” — and you lose context: without knowing what “it” is, the fragment is useless, and its embedding can't capture the meaning the surrounding text held. The sweet spot is a chunk containing one complete idea with enough context to stand on its own, often a paragraph or short section. Finding that size for your content is one of the highest-impact tuning decisions in a RAG system, because it directly controls whether retrieval hands the model a clear, self-contained passage or a blurry or fragmentary one.

3 · Overlap keeps ideas whole

A clever refinement solves a boundary problem. If you chop a document at hard edges, you'll inevitably split some idea down the middle — the end of one chunk plus the start of the next form a key sentence, but neither chunk contains the whole thing, so neither embeds it well and retrieval misses it. The fix is overlap: let consecutive chunks share a little edge, each repeating a bit of the previous chunk's ending. Then any idea near a boundary is fully contained in at least one chunk. It's like tearing a document into strips but leaving a little of each strip's text on the next — nothing important gets cut in half. A modest overlap costs a little extra storage but noticeably improves retrieval reliability, especially where ideas flow across paragraph breaks. It's a small setting with an outsized effect on catching passages that boundaries would otherwise slice apart.

4 · Chunk with the structure, not against it

How you choose where to cut matters as much as size. The naive approach splits every fixed number of characters — simple, but it slices mid-sentence and mid-idea, creating exactly the fragments to avoid. The far better approach is to chunk with the document's natural structure: split on headings, paragraphs, and sections, because those are where the author already separated ideas. A section under one heading is usually about one topic — a natural, self-contained chunk. Code has natural seams too: functions, classes, files. Respecting this structure means your chunks line up with real conceptual boundaries instead of arbitrary character counts, making every chunk cleaner and every embedding sharper. So before reaching for a blunt character-count splitter, look at how your documents are already organised — the author usually chunked the ideas for you, and you just follow their seams.

Overlapping chunks that follow the document's structure Three chunks split on paragraph boundaries, each sharing a small overlap with its neighbour. chunk 1 chunk 2 chunk 3 split on real boundaries · overlap the edges · nothing cut in half
Figure 2. Chunks that follow the document's seams and overlap slightly keep every idea whole and findable.

5 · Indexing: the organised, searchable store

Once documents are chunked, indexing turns them into something searchable. You run every chunk through the embedding model to get its point on the map, then store all those points — along with the original text — in a vector database. That organised, searchable store is your index: the map of meaning from the embeddings topic, now populated with your real content. Building the index is a one-time upfront cost; afterwards, every query is a fast “what's nearest?” lookup against it. It's the finished library, catalogued and ready — where retrieval, and therefore RAG, actually happens. Everything before this step was preparation; the index is the product of that preparation and the thing your app searches at answer-time.

6 · Metadata: labels that make chunks useful

A good index keeps more than raw points — it keeps metadata beside each chunk: which document and section it came from, its date, and who's allowed to see it. These labels do real work. Source and section let you show citations, so answers point back to where they came from. Dates let you prefer current material over stale. And permissions let you filter retrieval to only what a given user is allowed to read — the same access-control discipline from earlier, applied so RAG can never surface a document someone shouldn't see. So the index isn't just a pile of vectors; it's your content — chunked, embedded, labelled, and organised — so retrieval is fast, accurate, and safe. Metadata is what turns a searchable heap into a trustworthy one.

7 · When RAG answers badly, look here first

Here's the practical payoff. When a RAG system gives disappointing answers, most people blame the model — but far more often the culprit is chunking. The model answered fine from what it was given; the problem is what it was given was a blurry, oversized chunk, a fragment missing its context, or a passage split across a boundary and never retrieved whole. So when RAG underperforms, look at your chunks first: pull up what actually got retrieved for a failing query and read it. Is it a clean, self-contained passage that answers the question? Often you'll immediately see the problem. Fixing chunking — adjusting size, adding overlap, respecting structure — frequently rescues a RAG system that seemed hopelessly inaccurate. The unglamorous prep work is where the quality actually lives.

The one sentence to remember

Retrieval is only as good as your chunks — so a chunk should hold one clear idea, with enough context to stand on its own, and be cut along the document's own seams.

8 · A simple test you can run this week

Chunk one document by hand

1. Take a long document you'd want an AI to answer questions from.
2. Split it along its headings and paragraphs — not by character count.
3. Read each chunk and ask: does it stand alone as one clear, complete idea?
4. Fix any that are too big, too small, or cut mid-idea — merge, split, or overlap.

The lesson: good chunks in, good answers out — retrieval quality starts here.

9 · Glossary — every term, spelled out

Chunk
A small piece of a document, ideally about one idea, that gets embedded and stored separately.
Chunking
Splitting documents into chunks before embedding them.
Overlap
Letting consecutive chunks share a little text at their edges so ideas near a boundary aren't cut in half.
Structure-aware splitting
Chunking along a document's natural seams — headings, paragraphs, sections — rather than by fixed character counts.
Index
The organised, searchable store of all embedded chunks — the map of meaning filled with your content.
Metadata
Labels kept with each chunk — source, date, permissions — used for citations, filtering, and access control.
Key takeaways

You embed pieces of documents, not whole documents — each chunk should be about one idea.
Chunk size is Goldilocks: too big blurs meaning, too small loses context.
Use overlap so ideas aren't cut at boundaries, and split along the document's real structure.
The index is your chunked, embedded, labelled content — and most RAG failures are chunking failures.

References

  1. Pinecone, Chunking Strategies for LLM Applications — practical guidance on chunk size and overlap. pinecone.io
  2. Microsoft, Chunk documents in vector search — structure-aware chunking in practice. learn.microsoft.com
  3. This guide’s Embeddings & Vector Search, Explained From Zero — what you turn chunks into.
  4. This guide’s RAG, Explained From Zero — why retrieval, and thus chunking, decides answer quality.