Implement a Text Chunker with Overlap
On this page
Problem statement
Write a chunker for a RAG pipeline. Part 1:
chunk_tokens(tokens, size, overlap)splits a list of tokens into windows of at mostsizetokens where consecutive windows shareoverlaptokens. Part 2:chunk_sentences(sentences, count_tokens, max_tokens, overlap_sentences)packs whole sentences into chunks of at mostmax_tokens, repeating the last sentence(s) of each chunk at the start of the next.
A practical coding task for AI application roles: simple on the surface, full of off-by-one traps, and a natural lead-in to chunking strategy (see Chunking Strategies for RAG).
Example

chunk_tokens([0..9], size=4, overlap=1) → [0,1,2,3] [3,4,5,6] [6,7,8,9]
chunk_tokens([0..9], size=4, overlap=0) → [0,1,2,3] [4,5,6,7] [8,9]
chunk_tokens([1,2], size=4, overlap=1) → [1,2]
Constraints (assumed)
- Documents up to about a million tokens.
count_tokensis supplied (a real tokenizer in production; word counts in the example).- Pure Python.
What this question tests
- Getting loop boundaries right (no lost or duplicated tail).
- Input validation (overlap ≥ size would never advance).
- Respecting semantic units (sentences) while enforcing a hard limit.
- Testing properties, not just one example.
Questions to ask first
- Is the limit in tokens or characters? (Tokens.)
- Must chunks respect sentence or paragraph boundaries? (Part 2: sentences.)
- What if one sentence is longer than the limit? (Split it.)
- Is a short final chunk acceptable? (Yes.)
- Should chunk metadata (offsets, titles) be returned? (Not here; a follow-up.)
Try it yourself first
You have the full problem. Sketch your answer, then sign up free to read the complete walkthrough.
- The complete design, flow by flow, with pictures
- Deep dives into the hard parts
- What mid, senior and staff candidates are expected to say
- Full company question lists
Free forever. No card needed.