# Agent Evaluation, Done Right

Canonical: https://metix.ai/research/agent-evaluation-done-right

Engineering note on evaluating production agent systems: research-style testing over unit testing, component/trajectory/outcome levels, golden sets, rule checks with calibrated LLM judges, online evaluation, and evaluation wired into CI.
Bug-free code and correct agent behavior are two separate claims; what keeps a production agent standing is the evaluation system behind it.
## FAQ

**Why can't agent systems be tested like traditional software?**

Agents are non-deterministic: the same input can produce different reasoning and sometimes a different conclusion, so a passing unit test proves nothing permanent. Evaluation should follow machine-learning research practice instead, with a held-out test set, a handful of metrics, and every change compared against a baseline across the whole distribution.

**What are the three levels of agent evaluation?**

Component level scores one atomic capability such as a resume parse or a retrieval. Trajectory level scores the full execution path, including tool choice and recovery after errors. Outcome level scores end-to-end task quality and business metrics. You need all three: outcomes tell you something broke, components and trajectories tell you where.

**When should you use an LLM judge instead of rule checks?**

Never use a model where a rule will do; rule checks are cheap and zero-variance, and they catch a large share of regressions. LLM judges handle subjective dimensions like tone or faithfulness, with concrete rubrics, pairwise comparisons for versions, and the judge itself validated against human labels.

**What should you monitor after an agent ships?**

Implicit user behavior is the most honest signal: how heavily users edit the agent's output, the acceptance rate of recommendations, the human-takeover rate, and mid-flow abandonment. Every prompt or model change should go through canary and A/B before ramping.