Document Ingestion for RAG: Parsing, Cleaning, Updating

Updated Nov 16, 2026 · 5 min read
On this page

    An internal assistant answers "what's our refund window?" with "30 days". Policy changed to 14 days two weeks ago. Investigation finds three causes: the updated page was never re-indexed because the nightly job only picked up new documents; an old PDF copy of the policy is still indexed; and the policy's pricing table was flattened into an unreadable line of numbers, so retrieval preferred the stale PDF. Nothing was wrong with the model or the retriever. Most RAG quality and safety problems start in ingestion: how documents are fetched, parsed, cleaned, chunked, permissioned and kept in sync.

    How it works

    Sources flow through connectors, parsing, cleaning, chunking with metadata, and embedding into the index.
    Sources flow through connectors, parsing, cleaning, chunking with metadata, and embedding into the index.

    1. Connect

    Connectors pull from each source (wiki, shared drive, ticketing system, code host) using its API. Two things matter beyond the content:

    • Change detection: webhooks or change feeds where the source offers them, polling with "modified since" otherwise, plus a periodic full reconciliation to catch missed events and deletions.
    • Permissions: the access list of every document (users, groups) is fetched and stored with it, so retrieval can filter per user. A RAG index without ACLs leaks documents to people who couldn't open them.

    2. Parse

    Common parsing failures and what layout-aware parsing keeps instead.
    Common parsing failures and what layout-aware parsing keeps instead.
    • HTML: strip navigation, footers and cookie banners; keep headings, lists, code and tables.
    • PDFs and slides: use layout-aware parsing so two-column text is read in the right order, tables become rows with headers (Markdown or HTML), and repeated headers and footers are removed. Scanned pages need OCR; mark them as lower confidence.
    • Images and diagrams: a vision model can write a description or extract text when the content matters.
    • Keep structure as metadata: title, heading path, page number, section, author, last-modified date, source URL.

    3. Clean

    Normalise whitespace and Unicode, drop empty and boilerplate-only sections, remove exact and near-duplicate documents (the same policy saved in three folders), and decide which copy is authoritative (newest, from the official space). Optionally detect and redact PII that shouldn't be retrievable.

    4. Chunk and enrich

    Split into chunks along structure (sections, paragraphs, table rows) and prefix each chunk with its document title and heading path, so "it costs $49" becomes "Pricing > Team plan: it costs $49" (see Chunking Strategies for RAG). Attach metadata for filtering: source, date, product, language, ACL.

    5. Embed and index

    Embed chunks, write vectors plus keyword (BM25) fields and metadata to the index, and record which embedding model version produced them.

    Keeping it in sync

    Hash each chunk, re-embed only those whose content changed, update permissions without re-embedding, and remove deleted documents.
    Hash each chunk, re-embed only those whose content changed, update permissions without re-embedding, and remove deleted documents.
    1. On each change event, re-parse and re-chunk the document.
    2. Compare each chunk's content hash with what's stored: embed only new or changed chunks.
    3. Permission changes update metadata only; no re-embedding needed.
    4. Delete chunks that no longer exist, and delete everything for a deleted document (tombstones).

    The included incremental_sync.py shows this on a refund policy: one edited fact costs one embedding call, narrowing access costs none, and deleting the document empties its entries from the index.

    Trade-offs and costs

    • Parsing quality vs cost: layout models and vision-based parsing are slower and pricier than plain text extraction; use them for PDFs and slides where it matters.
    • Chunk identity: position-based chunk IDs are simple but an insertion near the top shifts every later chunk; content-defined IDs (hash of the chunk's heading path and text) re-embed less.
    • Freshness vs load: real-time sync costs API quota and compute; batch is cheaper but stale. Set a freshness target per source (minutes for policies and tickets, a day for archives).
    • Re-embedding everything when you change the embedding model: plan it as a background backfill into a new index, then switch.

    In production

    • Per stage, track documents in and out, parse failures by file type, chunks per document, embedding errors, and freshness lag (time from source change to searchable).
    • Treat deletes and permission changes as correctness and security events with their own, tighter targets.
    • Keep a dead-letter queue for documents that fail to parse and review it.
    • Store the raw document and parsed output so you can re-chunk or re-embed without refetching.
    • Spot-check retrieved chunks for real user questions; ingestion bugs show up there first.

    When to reach for it

    Any RAG system over real company content, and "design enterprise search / a RAG platform" interviews. If retrieval quality is poor, inspect ingestion output before tuning embeddings or rerankers.

    Example

    The refund assistant from the opening is fixed with: webhook-based sync from the wiki plus hourly reconciliation (freshness lag from up to 24 hours to under 5 minutes); deduplication that keeps only the official wiki version of each policy and drops PDF exports; layout-aware PDF parsing that turns pricing tables into rows; heading-path prefixes on every chunk; and per-user ACL filtering at query time. On a set of 200 real employee questions, correct-answer rate goes from 71% to 88% without touching the model. (Example numbers.)

    Common mistakes

    • Indexing once and never syncing, or syncing additions but not edits and deletions.
    • Ignoring permissions, so retrieval shows people documents they can't open.
    • Flattening tables and multi-column PDFs into unreadable text.
    • Indexing duplicates and stale copies next to the authoritative version.
    • Losing headings and titles, so chunks lack context.
    • No ingestion metrics, so failures are discovered through wrong answers.

    Say this in the interview

    "Ingestion is a pipeline: connectors with change feeds and permissions, layout-aware parsing that keeps reading order, tables and headings, cleaning and de-duplication with one authoritative copy, structure-aware chunks prefixed with their title path and rich metadata, then embedding and indexing with ACLs. To stay fresh, I'd process change events, hash chunks to re-embed only what changed, update permissions as metadata and propagate deletes quickly, with periodic reconciliation. I'd track freshness lag and parse failures per source."

    Interview questions on this

    1. How do you keep a RAG index in sync with a wiki? Webhooks or change feeds, chunk hashing, re-embed only changes, tombstones for deletes, reconciliation.
    2. How do you enforce document permissions in RAG? Store ACLs per chunk, filter at query time by the user's identity and groups.
    3. How would you handle tables in PDFs? Layout-aware parsing into rows with headers; keep each row with its header as a chunk or summary.
    4. You change embedding models. What happens to the index? Backfill a new index in the background, compare quality, then switch atomically.
    5. What metrics show ingestion health? Freshness lag, parse failure rate, docs in and out per stage, chunks per doc.
    6. Two copies of a policy disagree. Which one does RAG use? Define authority rules (source, date) and dedupe at ingestion.

    Used in

    • AI Engineering: Chunking Strategies for RAG; RAG End to End; Embeddings: Turning Text into Vectors; Hybrid Search and Reranking; Your RAG Answers Are Wrong: How Do You Debug It?; Building a Training Data Pipeline.
    • AI System Design: Design Enterprise RAG; Design a Document Processing Pipeline; Design Perplexity.