Evaluating Agents

Updated Nov 16, 2026 · 4 min read
On this page

    A refund agent's final message reads "Done! Refund issued for order 4521." The team's eval checks the final message with an LLM judge and scores it as a pass. Looking at the trace tells a different story: 38 steps for a 5-step task, a refund issued to the wrong order first, and the refund policy never checked. Agents act, so evaluating them means checking what they did, not only what they said.

    How it works

    A pleasant final message can hide a costly or harmful run.
    A pleasant final message can hide a costly or harmful run.

    What to grade

    Outcome first, then how the agent got there, then what it cost.
    Outcome first, then how the agent got there, then what it cost.
    1. Outcome: is the task actually done? Check the end state with code wherever possible: the right database row exists, the file changed correctly, the tests pass, the email was sent to the right person (in a test environment). This is the most trustworthy signal.
    2. Trajectory: did it get there the right way? Right tools in a sensible order, required checks done (policy looked up before refund), no forbidden actions, no repeated or wasted steps. Check rule-like properties with code over the trace; use an LLM judge with a rubric for softer ones ("asked a clarifying question when the order ID was missing").
    3. Cost and time: steps, tokens, money, wall-clock time, and how often it escalated to a human.

    Build a harness

    Each task starts from a known environment; a checker inspects what changed.
    Each task starts from a known environment; a checker inspects what changed.
    • Tasks: realistic requests with a defined starting state (a seeded test database, a repository snapshot, a mock mailbox) and success criteria.
    • Sandbox: the agent runs against test copies or mocks of its tools, never production.
    • Checker: code that inspects the end state and the trace, plus judge-based checks where needed.
    • Repeats: agents are not deterministic, so run each task several times.
      • pass@k: succeeds at least once in k tries; shows what the agent can do.
      • pass^k: succeeds in all k tries; shows how reliable it is. With 80% per try, pass^3 is about 0.8³ ≈ 51%. Users experience reliability, not capability.

    Where tasks come from

    Start with 20–50 tasks written from real requests and known failures, then grow from production: every incident or escalation becomes a task. Include edge cases (missing information, ambiguous requests, tool errors) and adversarial ones (injected instructions in data). Benchmarks from the community are useful for comparing models, but your own tasks decide whether your agent works.

    Trade-offs and costs

    • Sandboxes take work to build and keep realistic; mocks that are too simple make evals pass while production fails.
    • Agent evals are slow and expensive: many steps per task, several repeats per task. Run a small core set on every change and the full set nightly or before releases.
    • Judges on trajectories can be inconsistent on long traces; prefer code checks, and calibrate judges against human review.
    • Overfitting: tuning prompts against the same tasks repeatedly; keep a held-out set.

    In production

    • Online signals: task completion, user corrections, escalations, undo actions, complaints.
    • Trace sampling: review a sample of real runs with a judge and humans; turn failures into eval tasks.
    • Regression gates: block releases when outcome success or reliability (pass^k) drops on the core set.
    • Failure modes: evals that only read the final message; tasks that no longer match production tools; nondeterminism hiding regressions when each task runs once.

    When to reach for it

    Before shipping any agent, and on every change to its prompt, model, tools or orchestration. For high-risk actions (money, data deletion, external messages), trajectory rules are as important as outcomes.

    Example

    The refund agent's new eval has 60 tasks in a seeded test environment. The checker verifies the right order got the right refund amount, that the policy tool was called before the refund tool, that no other order changed, and that the run stayed under 10 steps. Each task runs 3 times. The old version scored 88% pass@3 but only 52% pass^3, and 30% of runs violated the "check policy first" rule. After making policy lookup part of the refund tool itself and adding argument validation, pass^3 rose to 85% with no rule violations (example numbers).

    Common mistakes

    • Grading only the final message.
    • Running each task once and treating the result as stable.
    • Evaluating against production systems or unrealistic mocks.
    • No cost or step budget in the scoring.
    • Never adding production failures to the task set.

    Say this in the interview

    "I evaluate outcome, trajectory and cost. Outcome is checked with code against the end state in a sandboxed environment; trajectory checks cover required steps, forbidden actions and wasted steps, with code first and a calibrated judge for softer criteria. Each task runs several times; I track pass@k for capability and pass^k for reliability, plus steps, tokens and time. Production incidents become new tasks, and a core set gates every release."

    Interview questions on this

    1. Why isn't the final answer enough to grade an agent? It can claim success while the state is wrong or the path was unsafe or wasteful.
    2. What's the difference between pass@k and pass^k? Any success in k tries vs success in all k tries; the second reflects user-facing reliability.
    3. How do you evaluate an agent that sends emails? In a sandbox with a mock mailbox; check recipients, content rules and that nothing else was sent.
    4. Where do eval tasks come from? Real requests, known failures, edge and adversarial cases, and production incidents.
    5. How do you keep agent evals affordable? A small core set per change, the full set less often, cheaper models for judging where calibrated.

    Used in

    • AI Engineering: Evals and LLM-as-Judge; Making Agents Reliable; Agent Safety; Online Evaluation and A/B Tests; LLM Observability and Tracing.
    • AI System Design: Design an Eval and Observability Platform; Design a Customer Support Agent.