Evals and LLM-as-Judge
On this page
A customer complains that the assistant's answers are too long. An engineer adds "be concise" to the prompt, checks the complaint's example, and ships. A week later, support notices the assistant now skips safety warnings and drops steps from setup instructions. Nobody saw it coming because nobody tested beyond the one example. Evals are the test suite for LLM features: a fixed set of cases and a way to score outputs, run on every change.
How it works

The pieces of an eval
- Dataset: inputs that represent real use. Start with 50–200 cases from logs, tickets and expert examples; include edge cases, adversarial inputs and past failures. Each case may have a reference answer or a list of properties a good answer must have.
- Scorer: a way to decide whether an output is good.
- Runner: runs the system (prompt, model, settings, retrieval) over the dataset and stores outputs and scores, so versions can be compared case by case.
Choosing the scorer

- Code checks for anything objective: exact match for labels, JSON schema validation, regex for required phrases, unit tests for generated code, length limits, "contains a citation". Fast, free and stable.
- LLM-as-judge for open-ended qualities: helpfulness, faithfulness to sources, tone, completeness. Two common forms:
- Rubric scoring: the judge reads the input, the output (and reference, if any) plus a rubric, and returns a score with a short reason.
- Pairwise comparison: the judge sees answers A and B and picks the better one; often more reliable than absolute scores for comparing two versions.
- Human review for high-stakes judgments and for checking the judge itself.
Making LLM-as-judge trustworthy

- Write a specific rubric: what earns a 1, a 3, a 5, with examples. Vague rubrics give noisy scores.
- Ask for the reason before the score, so the judgement is grounded.
- Calibrate: have people label a few hundred outputs, run the judge on the same ones, and measure agreement. Fix the rubric until agreement is acceptable.
- Control for known biases: position bias (swap A and B and average), length bias (instruct to ignore length, or compare at similar lengths), self-preference (use a judge from a different model family than the one being judged when possible).
- Pin the judge model version; a new judge means re-calibrating.
Trade-offs and costs
- Dataset quality beats size. 100 well-chosen cases catch more regressions than 5,000 random ones.
- Judge cost adds up: every eval run is a model call per case. Use small judges for simple properties and big ones only where needed.
- Over-fitting to the eval: if you tune prompts only against a fixed set, you may overfit; keep a held-out set and refresh with new production failures.
- Scores aren't the goal. A metric rising while users complain means the metric measures the wrong thing.
In production
- CI gate: run the eval on every prompt, model or retrieval change; block merges that drop key metrics beyond a threshold.
- Online evals: run cheap judges on a sample of live traffic, track scores over time, and alert on drops (see Online Evaluation and A/B Tests).
- Close the loop: every production failure becomes a new eval case.
- Failure modes: a judge drifting after its provider updates it; flaky scores from high temperature (judge at temperature 0); an eval set that no longer looks like real traffic.
When to reach for it
From the first prototype. Even 30 cases with simple checks beat demo-driven development. Evals are also how you justify model switches, prompt changes and cost cuts.
Example
A support assistant's eval set has 180 cases: 120 from real tickets, 40 known failures and 20 adversarial prompts. Scorers: JSON validity and "includes a help link" by code; faithfulness and helpfulness by a judge with a 1–5 rubric, calibrated at 86% agreement with two support leads on 200 labelled answers. The "be concise" change scored +0.4 on brevity but −0.6 on completeness and failed 9 safety-warning cases, so it was rewritten as "be concise, but always keep warnings and all setup steps", which passed.
Common mistakes
- Testing on the example that prompted the change and nothing else.
- Using an LLM judge where a simple code check would do.
- Never measuring judge agreement with people.
- Judging with the same model that produced the answers, without checking self-preference.
- Letting the eval set go stale.
Say this in the interview
"Evals are a test suite: a dataset of real and adversarial cases, a scorer, and a runner that compares versions case by case. I use code checks for anything objective, LLM-as-judge with a specific rubric for open-ended quality, and humans for high-stakes cases and for calibrating the judge. I control judge biases (position, length, self-preference), run evals in CI on every change, and turn production failures into new cases."
Interview questions on this
- How do you build an eval set from scratch? Sample real inputs, add known failures and edge cases, label expected properties, and grow it with production failures.
- Rubric score or pairwise comparison? Pairwise for comparing two versions; rubric for tracking absolute quality over time.
- How do you know the judge is right? Measure agreement with human labels on a sample and re-check when the judge or rubric changes.
- What is position bias, and how do you handle it? Judges favour the first answer shown; evaluate both orders and combine.
- How do you stop overfitting prompts to the eval set? Keep a held-out set, refresh with new cases, and confirm with online metrics.
Used in
- AI Engineering: Evaluating a RAG System; Evaluating Agents; Online Evaluation and A/B Tests; How Do You Ship a Prompt Change Safely?; Evaluating a Fine-tuned Model.
- AI System Design: Evals and LLM-as-judge; Design an Eval and Observability Platform.