# Chunking documents

> Split documents into pieces small enough to match a question and big enough to answer it.

Concept · Embeddings & retrieval · https://levelcode.org/concepts/chunking

Retrieval works on **chunks**, not whole documents. Embed a 40-page manual as one vector and its meaning blurs into "this is a manual," so it matches every question weakly. Split it into passages and the one about refunds can match a refund question precisely. **Chunking** decides where those splits go, and it's often the single biggest factor in [RAG](https://levelcode.org/roadmaps/ai-engineer/rag) quality.

## The tension

- **Too small** (a sentence): it matches precisely but lacks context. "It must be filed within 30 days" doesn't say what "it" is.
- **Too large** (several pages): the [embedding](https://levelcode.org/concepts/embedding-similarity) gets diluted, irrelevant text fills the [context window](https://levelcode.org/concepts/context-windows), and costs go up.

A common starting point is **200–800 tokens per chunk with 10–20% overlap**, so a sentence cut at a boundary appears whole in at least one chunk. Then tune on your own questions.

## Strategies

| Strategy | How | Good for |
|---|---|---|
| Fixed size | Every *n* tokens, with overlap | Uniform plain text; a baseline |
| Recursive | Split on paragraphs, then sentences, then words, until chunks fit | Most prose; a sensible default |
| Structure-aware | Split on headings, sections, list items, table rows, functions | Markdown, HTML, docs, code |
| Semantic | Split where the topic shifts (embedding similarity drops between sentences) | Long unstructured text |

**Use the structure the document already has.** A Markdown section under `## Refund policy` is a better chunk than an arbitrary 500-token window that starts halfway through it.

## Make each chunk stand on its own

- **Prepend context:** the document title and heading path, such as `Employee handbook › Leave › Parental leave:`. This alone often improves retrieval noticeably. Some pipelines go further and have a model write a one-sentence summary of where the chunk fits, then embed that with the chunk.
- **Store metadata** with each chunk (source, section, date, access permissions) so you can filter and cite.
- **Keep tables and code blocks intact.** A table split down the middle loses its headers.

## Retrieve small, return big

Some pipelines index small chunks for precise matching but give the model the **surrounding section**, or neighbouring chunks, once a chunk matches. You get precision from the small piece and context from the large one.

Whatever you choose, judge it by retrieval [evals](https://levelcode.org/concepts/llm-as-judge): for a set of real questions, does the right chunk land in the top *k*? Adding [hybrid search and reranking](https://levelcode.org/concepts/hybrid-search-reranking) on top recovers much of what imperfect chunking misses.