Concept · Embeddings & retrieval
Chunking documents
Split documents into pieces small enough to match a question and big enough to answer it.
Retrieval works on chunks, not whole documents. Embed a 40-page manual as one vector and its meaning blurs into "this is a manual," so it matches every question weakly. Split it into passages and the one about refunds can match a refund question precisely. Chunking decides where those splits go, and it's often the single biggest factor in RAG quality.
The tension
- Too small (a sentence): it matches precisely but lacks context. "It must be filed within 30 days" doesn't say what "it" is.
- Too large (several pages): the embedding gets diluted, irrelevant text fills the context window, and costs go up.
A common starting point is 200–800 tokens per chunk with 10–20% overlap, so a sentence cut at a boundary appears whole in at least one chunk. Then tune on your own questions.
Strategies
| Strategy | How | Good for |
|---|---|---|
| Fixed size | Every n tokens, with overlap | Uniform plain text; a baseline |
| Recursive | Split on paragraphs, then sentences, then words, until chunks fit | Most prose; a sensible default |
| Structure-aware | Split on headings, sections, list items, table rows, functions | Markdown, HTML, docs, code |
| Semantic | Split where the topic shifts (embedding similarity drops between sentences) | Long unstructured text |
Use the structure the document already has. A Markdown section under ## Refund policy is a better chunk than an arbitrary 500-token window that starts halfway through it.
Make each chunk stand on its own
- Prepend context: the document title and heading path, such as
Employee handbook › Leave › Parental leave:. This alone often improves retrieval noticeably. Some pipelines go further and have a model write a one-sentence summary of where the chunk fits, then embed that with the chunk. - Store metadata with each chunk (source, section, date, access permissions) so you can filter and cite.
- Keep tables and code blocks intact. A table split down the middle loses its headers.
Retrieve small, return big
Some pipelines index small chunks for precise matching but give the model the surrounding section, or neighbouring chunks, once a chunk matches. You get precision from the small piece and context from the large one.
Whatever you choose, judge it by retrieval evals: for a set of real questions, does the right chunk land in the top k? Adding hybrid search and reranking on top recovers much of what imperfect chunking misses.