Your RAG Answers Are Wrong: How Do You Debug It?

Free walkthrough Updated Nov 16, 2026 · 3 min read
On this page

    Problem statement

    "Our internal HR assistant uses RAG over the policy wiki. Employees report wrong answers, for example that contractors get the 2026 learning budget, which they don't. How would you debug this?"

    Interviewers ask this to see whether you debug systematically, stage by stage, or jump straight to "use a bigger model".

    Why it happens

    Check the stages in order. The first "no" is where to fix.
    Check the stages in order. The first "no" is where to fix.

    A RAG answer passes through several stages, and each can fail silently:

    • Index: the document was never ingested, ingestion failed on its format, or an old version is still indexed alongside the new one.
    • Chunking: the key sentence was split, or a table lost its header.
    • Retrieval: the right chunk exists but the query embedding is closer to other text; exact terms like "contractor" or "2026" are lost in vector search.
    • Ranking: the right chunk is retrieved but ranked below noise or a stale version.
    • Generation: the model ignores the passage, mixes two versions, or answers from its own general knowledge.
    • Sometimes the label is wrong: the "correct" answer used for comparison is outdated.

    Options

    Weakchange the model or the prompt first

    Swap to a bigger model or add "be accurate" to the prompt, then try the reported question again. If retrieval never returned the right passage, no model can fix it, and checking one question by hand tells you nothing about the rest.

    Goodlook at traces and fix the obvious stage

    Log every request with retrieved chunk IDs, scores and the final prompt. Read the traces for the reported failures; you'll often see the cause at once (as in the trace below), fix it, and re-test those questions.

    Example trace: the stale 2025 policy outranks the 2026 one, and the model answers from it.
    Example trace: the stale 2025 policy outranks the 2026 one, and the model answers from it.
    Strongbuild a test set and measure each stage
    • Collect failures from reports and logs; add similar questions and a sample of questions that work, to catch regressions. Label the gold document and chunk for each.
    • Measure per stage: document present and current (an index check), retrieval recall@50, recall@5 after reranking, faithfulness and correctness of answers.
    • Fix by stage:
      • Index: re-ingest, delete superseded versions, add "effective date" metadata and filter or boost by it.
      • Chunking: structure-aware chunks with headings; keep tables intact.
      • Retrieval: hybrid search so exact terms count; query rewriting for vague questions; metadata filters (region, employee type).
      • Ranking: a cross-encoder reranker; more candidates.
      • Generation: instruct to prefer the most recent effective date, cite sources, and abstain when sources conflict.
    • Re-run the full set after each change, and add the set to CI so the bug can't come back.

    How to measure it

    • Index coverage and freshness: share of gold documents present in their current version; ingestion lag.
    • Retrieval recall@k before and after reranking.
    • Answer correctness and faithfulness on the labelled set (judge plus spot checks).
    • Conflict handling: on questions where old and new versions exist, how often the newest is used.
    • Online: wrong-answer reports per 1,000 questions; thumbs-down rate by topic.

    Example answer flow

    "I'd pull the reported questions and 50 similar ones, label the correct policy chunk, and run them with tracing. In the contractor case, the trace shows both the 2025 and 2026 policies retrieved, with 2025 ranked first, so this is a freshness and ranking problem. I'd remove superseded versions at ingestion, add effective-date metadata with a recency boost, add hybrid search so 'contractor' and '2026' match exactly, and tell the model to prefer the latest effective date. Then I'd check recall@5 and correctness on the whole set, not just the reported question."

    Follow-up questions

    1. How do you label gold chunks quickly for a few hundred questions?
    2. Retrieval recall is fine but answers are still wrong. What next?
    3. How would you handle two policies that legitimately apply to different groups?
    4. How do you keep the index fresh when the wiki changes every day?
    5. How do you prevent the same bug from coming back?
    6. What would you log for every request, and what about privacy?

    Related

    • Concepts: RAG End to End; Chunking Strategies; Hybrid Search and Reranking; Evaluating a RAG System; Document Ingestion for RAG.
    • Practice: How Would You Reduce Hallucinations?
    • AI System Design: Design Enterprise RAG.