Chunking Strategies for RAG
On this page
A RAG assistant is asked "How long is the refund window?" The policy clearly says 30 days, yet the assistant answers "the policy doesn't specify". The cause sits in the index: the sentence was split across two chunks. The first chunk mentions the refund window but not the number; the second has "30 days" but no mention of refunds, so neither is a good match on its own. How you cut documents decides what retrieval can find, and it is one of the cheapest things to fix in a RAG system.
How it works

What a good chunk looks like
A chunk is the unit that gets embedded, retrieved and pasted into the prompt. A good one:
- makes sense on its own (a reader could answer something from it without the rest of the document);
- is about one topic, so its embedding isn't a blurry average of several;
- carries its context: the document title, section heading and key metadata travel with it.
Three common strategies

- Fixed size with overlap. Split every N tokens (for example 300–500) and repeat the last 10–20% of each chunk at the start of the next. Simple and predictable; the overlap reduces, but doesn't remove, broken sentences.
- By structure. Split on the document's own boundaries: headings, sections, paragraphs, list items, table rows, code functions. Then merge small pieces or split large ones to stay within a size range. Works very well for docs, wikis and code.
- By meaning (semantic). Embed sentences and start a new chunk where the topic shifts (similarity between neighbouring sentences drops). Helps with long unstructured text like transcripts, at the cost of an extra embedding pass.
Enrich the chunk
Prefix each chunk with where it came from, for example Refund policy > Timelines: plus the text. Some pipelines also ask a model to write one sentence of context per chunk ("This section describes refund timing for online orders") before embedding. Both make short chunks much easier to match.
Search small, read big

Parent–child chunking indexes small chunks (around 100–200 tokens) for matching, but when one matches, the whole parent section (around 1,000 tokens) is sent to the model. A related trick, sentence windows, returns a few sentences around the matching one.
Special content
- Tables: keep header rows with every chunk of rows, or convert each row to a sentence ("Plan Pro: 20 seats, $49 per month").
- Code: split by function or class, keeping the file path and signature.
- PDFs and slides: extract by layout (headings, columns) before chunking; plain text extraction often scrambles reading order (see Document Ingestion for RAG).
Trade-offs and costs
| Chunk size | Good | Bad |
|---|---|---|
| Small (~100–200 tokens) | Precise matches, cheap prompts | Missing context, more chunks to store |
| Medium (~300–600) | Usual sweet spot for prose | Still splits some answers |
| Large (~1,000+) | Full context | Blurry embeddings, expensive prompts |
These ranges are common starting points, not rules. Overlap and parent–child add storage. Semantic chunking adds compute at indexing time.
In production
- Tune with an eval set: measure retrieval recall@k for a few chunking settings on real questions before picking one (see Evaluating a RAG System).
- Re-chunking means re-embedding: chunk settings are part of the index version; plan migrations.
- Stable chunk IDs (document ID + section path) make updates and deletes possible when a document changes.
- Failure modes: answers split across boundaries; headers separated from their tables; boilerplate (menus, footers) chunked and retrieved; very short chunks matching everything vaguely.
When to reach for it
Every RAG system needs a chunking choice. Revisit it when retrieval misses answers that exist in the documents, when retrieved text lacks context, or when prompts get long and expensive.
Example
A help centre used fixed 1,000-token chunks; retrieval recall@5 on 200 test questions was 0.68. Switching to heading-based chunks of 200–500 tokens, each prefixed with "Article title > Section", raised recall@5 to 0.81. Adding parent–child (search 150-token children, return the parent section) raised it to 0.87, and answers improved because the model saw whole sections. Example numbers; the point is that each change was measured.
Common mistakes
- Choosing a chunk size once and never measuring.
- Dropping headings and titles, leaving chunks with no context.
- Splitting tables away from their headers.
- Indexing boilerplate like navigation and cookie banners.
- Forgetting that chunk changes require re-embedding the corpus.
Say this in the interview
"Chunks are the unit of retrieval, so each should be self-contained, on one topic, and carry its title and section path. I'd default to structure-aware chunks of a few hundred tokens with light overlap, keep tables and code intact, and consider parent–child: search small chunks, send the parent section. Then I'd tune size and strategy by measuring retrieval recall on a real question set."
Interview questions on this
- How do you pick a chunk size? Start around a few hundred tokens and compare recall@k on real questions for two or three settings.
- Why add overlap? So sentences and facts near a boundary appear whole in at least one chunk.
- How do you chunk tables? Keep headers with each row group, or turn rows into self-contained sentences.
- What is parent–child retrieval? Match on small chunks, return their larger parent section to the model.
- What happens when you change chunking? Every document must be re-chunked and re-embedded; treat it as a new index version.
Used in
- AI Engineering: RAG End to End; Embeddings; Document Ingestion for RAG; Evaluating a RAG System; Implement a Text Chunker with Overlap (ML coding).
- AI System Design: Design Enterprise RAG; Design a Document Processing Pipeline.