Evaluating a RAG System
On this page
A team has two versions of its RAG pipeline: version 1 with large chunks, version 2 with smaller chunks and a reranker. Both "look good" on the five questions everyone tries in demos. Version 2 costs a bit more latency. Should it ship? Without numbers, the decision is a matter of opinion, and the next regression will reach users before anyone notices. Evaluating RAG means measuring each stage on a fixed set of real questions, so changes are decided by data.
How it works

Build a test set
- Collect real questions: from logs, support tickets, search queries, or subject-matter experts. Aim for 100–300 to start, covering common questions, rare ones, multi-part ones and questions the documents can't answer.
- Label the gold passages (which chunk or document contains the answer) and, where possible, a reference answer.
- Add synthetic questions if real ones are scarce: have a model write questions from sampled passages, then filter out trivial or ambiguous ones. Synthetic sets are useful but easier than real traffic; don't rely on them alone.
- Version the set and keep it stable, adding new cases from production failures.
Measure each stage

Retrieval (needs gold passages):
- Recall@k: share of questions whose gold passage is in the top k.
- MRR (mean reciprocal rank): the average of 1 / rank of the gold passage, rewarding it for ranking high.
- Precision@k / context precision: how much of what you retrieved is actually relevant.
def recall_at_k(retrieved, gold, k):
"""Share of questions whose gold passage appears in the top k."""
hits = sum(1 for r, g in zip(retrieved, gold) if g in r[:k])
return hits / len(gold)
def mrr(retrieved, gold):
"""Mean of 1/rank of the gold passage (0 when missing)."""
total = 0.0
for r, g in zip(retrieved, gold):
total += 1.0 / (r.index(g) + 1) if g in r else 0.0
return total / len(gold)
retrieved = [["p7", "p2", "p9"], ["p4", "p1", "p3"], ["p5", "p6", "p8"]]
gold = ["p2", "p4", "p1"]
print(round(recall_at_k(retrieved, gold, 3), 2), round(mrr(retrieved, gold), 2)) # 0.67 0.5
Generation (usually scored by an LLM judge, checked against human labels):
- Faithfulness (groundedness): are the answer's claims supported by the retrieved passages?
- Answer relevance: does it actually answer the question?
- Correctness: does it match the reference answer?
- Abstention: for unanswerable questions, does it say so instead of inventing?
End to end, online: thumbs up/down, follow-up questions that signal confusion, escalations to humans, task completion.
Compare versions

Run every candidate version against the same set, compare all metrics plus latency and cost, and look at the individual cases that changed, not just averages.
Trade-offs and costs
- Labelling takes time; gold passages for 200 questions can take a few days of expert work, but it is the most valuable asset in the project.
- LLM judges are cheap and fast but biased (toward longer answers, toward their own model family, toward the first option shown). Calibrate them against a few hundred human labels (see Evals and LLM-as-Judge).
- Offline sets drift from real traffic; refresh with new production failures.
- Too many metrics blur decisions; pick a primary metric per stage and a few guardrails (latency, cost, abstention).
In production
- Run evals in CI for every change to prompts, chunking, embedding model, retrieval settings or LLM.
- Sample live traffic for continuous scoring (faithfulness via judge, user feedback), and alert on drops.
- Log everything needed to replay: query, retrieved IDs, scores, prompt version, answer.
- Failure modes: a test set too easy to show differences; judges silently changing when their model updates; gold labels pointing at outdated documents.
When to reach for it
Before the first launch, and on every change after it. It also settles debates: "Should we switch embedding models?" becomes "Run the eval."
Example
The team built 200 questions from support logs, with gold passages labelled by two support leads, plus 20 unanswerable questions. Version 2 raised recall@5 from 0.71 to 0.86 and correctness from 0.64 to 0.77, abstained correctly on 18 of 20 unanswerable questions (version 1: 11), and cost less per answer because it sent fewer tokens, at +0.4 s p95 latency. They shipped version 2 and added the 15 questions both versions still failed as a focus list for the next iteration.
Common mistakes
- Only measuring the final answer, so you can't tell retrieval failures from generation failures.
- No unanswerable questions in the set, so hallucination never shows.
- Trusting an uncalibrated LLM judge.
- Changing the test set and the system at the same time.
- Looking at averages only, missing that a change broke one important question type.
Say this in the interview
"I'd build a test set of real questions with gold passages and reference answers, including unanswerable ones. Then I'd measure retrieval with recall@k and MRR, and generation with faithfulness, relevance and correctness, scored by an LLM judge calibrated against human labels. Every change runs against the same set in CI, I compare metrics plus latency and cost, read the cases that changed, and keep scoring a sample of live traffic."
Interview questions on this
- How do you tell whether a bad answer is a retrieval or a generation problem? Check if the gold passage was retrieved; if yes, it's generation; if no, retrieval.
- What is faithfulness and how do you measure it? Whether each claim is supported by the retrieved context; split into claims and verify with a judge or NLI model.
- How do you get a test set without labelled data? Mine logs and tickets, ask experts for a small seed, and add filtered synthetic questions generated from passages.
- Why include unanswerable questions? To measure whether the system abstains instead of hallucinating.
- How do you trust an LLM judge? Compare it to human labels on a sample, measure agreement, fix the rubric, and re-check when the judge model changes.
Used in
- AI Engineering: Evals and LLM-as-Judge; RAG End to End; Hybrid Search and Reranking; Online Evaluation and A/B Tests; Your RAG Answers Are Wrong (practical question).
- AI System Design: Evals and LLM-as-judge; Design an Eval and Observability Platform; Design Enterprise RAG.