Implement a Text Chunker with Overlap

Free with account Updated Nov 16, 2026 · 4 min read
On this page

    Problem statement

    Write a chunker for a RAG pipeline. Part 1: chunk_tokens(tokens, size, overlap) splits a list of tokens into windows of at most size tokens where consecutive windows share overlap tokens. Part 2: chunk_sentences(sentences, count_tokens, max_tokens, overlap_sentences) packs whole sentences into chunks of at most max_tokens, repeating the last sentence(s) of each chunk at the start of the next.

    A practical coding task for AI application roles: simple on the surface, full of off-by-one traps, and a natural lead-in to chunking strategy (see Chunking Strategies for RAG).

    Example

    With size 4 and overlap 1, each window starts 3 tokens after the previous one.
    With size 4 and overlap 1, each window starts 3 tokens after the previous one.
    chunk_tokens([0..9], size=4, overlap=1) → [0,1,2,3] [3,4,5,6] [6,7,8,9]
    chunk_tokens([0..9], size=4, overlap=0) → [0,1,2,3] [4,5,6,7] [8,9]
    chunk_tokens([1,2],  size=4, overlap=1) → [1,2]

    Constraints (assumed)

    • Documents up to about a million tokens.
    • count_tokens is supplied (a real tokenizer in production; word counts in the example).
    • Pure Python.

    What this question tests

    • Getting loop boundaries right (no lost or duplicated tail).
    • Input validation (overlap ≥ size would never advance).
    • Respecting semantic units (sentences) while enforcing a hard limit.
    • Testing properties, not just one example.

    Questions to ask first

    1. Is the limit in tokens or characters? (Tokens.)
    2. Must chunks respect sentence or paragraph boundaries? (Part 2: sentences.)
    3. What if one sentence is longer than the limit? (Split it.)
    4. Is a short final chunk acceptable? (Yes.)
    5. Should chunk metadata (offsets, titles) be returned? (Not here; a follow-up.)