Agent Evaluations: A Practical Guide to Metrics, Testing and Release Criteria
Agent evaluations test whether an AI agent completes the right task, respects its limits and leaves the system in the expected state. A convincing answer is not enough. If an agent says it updated a customer record, the evaluation should check the record, the permission decision and whether an unintended change occurred.
This guide explains how to build a practical pre-release evaluation plan: choose representative tasks, combine automated checks with human review, measure failures and set release criteria. It focuses on acceptance and regression testing. Use our separate AI agent security guide for access controls and threat protection.
Define the outcome and the limits together
Write a task specification before selecting an evaluator. Include the initial state, permitted tools, expected result, forbidden actions and what the agent should do when information is missing. Keep the result independent of the agent’s own claim of success.
For a customer-support agent, “answer politely” is an incomplete requirement. A useful task might require it to find the correct order, explain an eligible return, and create a draft request without issuing a refund. Correct wording cannot compensate for an unauthorised payment.
Anthropic’s guide to agent evaluations distinguishes the interaction trace from the final state. That distinction matters for agents that call tools: inspect observable outcomes, not just the final response. Automated evaluation makes checks repeatable; it does not make the underlying model deterministic.
Build a representative evaluation dataset
Start with examples from the workflow the agent will actually handle. Remove personal information and credentials, obtain the appropriate permission to use the data, and keep test systems separate from production. Synthetic examples can cover gaps, but should not be the only evidence of real-world performance.
- Routine successful tasks with clear inputs.
- Incomplete or contradictory information that should trigger clarification.
- Out-of-scope requests that require refusal or escalation.
- Tool timeouts, unavailable records and partial execution.
- Attempts to cross account or permission boundaries.
- Tasks whose wording differs while their underlying intent is the same.
Tag cases by workflow, risk and expected behavior. Keep a held-out set separate from examples used to adjust prompts. Otherwise, apparent improvement may reflect repeated tuning to the test set rather than better performance on unfamiliar work.
There is no universal minimum number of examples or required percentage of failures. A small initial suite can expose obvious defects, but cannot establish reliability for rare, serious events. Expand coverage according to task diversity, observed incidents and the decision the test must support.
Choose an agent evaluator for each kind of evidence
An agent evaluator is not necessarily another agent or a language model. Use ordinary code when the requirement can be checked directly. Add rubric-based review for qualities that cannot be captured by exact equality.
| Method | Useful for | Limit to manage |
|---|---|---|
| Deterministic checks | Schema, required fields, database state, allowed tool calls and arithmetic | A valid format does not establish factual or business correctness |
| Model-based evaluation | Comparing answers against an explicit rubric or reference evidence | Judges can be inconsistent or biased; calibrate against reviewed examples |
| Human review | Ambiguous cases, domain judgment and checking evaluator mistakes | Reviewers need shared criteria; disagreements require resolution |
Use a stable rubric with concrete examples of acceptable and unacceptable output. Ask a model-based judge to assess one defined criterion at a time and provide a short evidence-based explanation. Treat its score as a measurement to validate, not an independent authority.
For retrieval-heavy tasks, distinguish retrieval quality from answer quality. Our RAG versus fine-tuning comparison explains why missing knowledge and inconsistent behavior require different interventions. Changing the model will not repair a missing or inaccessible source document.
Use AI evaluation metrics that expose failure
| Metric | Definition | Why it matters |
|---|---|---|
| Accepted-task rate | Attempts meeting all required outcome and safety conditions ÷ attempts | Prevents fluent but incomplete answers being counted as success |
| Critical failure count | Observed violations of a named high-impact requirement | Must remain visible instead of disappearing inside an average score |
| Escalation quality | Whether the agent escalates when required and supplies useful context | Distinguishes safe uncertainty from silent failure or unnecessary handoff |
| Human correction effort | Time spent correcting or finishing the work | Shows the operational burden behind apparent automation |
| Latency | Time to an accepted result, including tool calls and retries | Reflects the user’s experience, not just model response time |
| Cost per accepted task | Total measured run costs ÷ accepted outcomes | Includes the economic effect of failed attempts |
Report these by task category as well as overall. For example, a suite containing mostly easy lookups can look healthy while the smaller record-update category repeatedly fails. Report the sample size with every rate and keep critical violations separate from quality scores.
Measure cost using the same workload definition as your AI agent cost model. An evaluation benchmark that omits retries or review will not predict operating cost well.
A worked test case: update an account owner safely
Illustrative engineering example. In a sandbox, create two customer accounts and give the requesting user access to only one. Ask the agent to change the owner of the accessible account. Supply an approved destination owner and require a confirmation step before writing.
- Check that the correct account was retrieved and the other account was not exposed.
- Check that the proposed change contains the exact account and owner identifiers.
- Verify that no write happens before the required confirmation.
- After confirmation, inspect the stored owner field rather than trusting the final message.
- Repeat with a denied permission, an ambiguous account name and a tool timeout.
- Check that retrying does not create duplicate side effects and that the final response describes what actually happened.
A schema check may confirm that an owner identifier has the right format. A state check establishes whether the intended record changed. A human reviewer can assess whether the explanation is clear and whether escalation supplied enough context. These checks answer different questions and should not replace one another.
Measure variation without overstating confidence
Run repeated trials when model variation could change the release decision. Record the model, settings, prompt version, tool versions, dataset version and starting state. Reset the environment between trials so a previous successful write does not make the next run appear easier.
Distinguish success across repeated attempts at one task from success across many different tasks. Trials can share failure causes and are not automatically independent. Five successful runs do not establish that a rare serious failure is acceptably unlikely.
Choose the number of cases and repetitions from the risk and precision required. If reporting confidence intervals, use a method appropriate to the sampling design and explain its assumptions. Report raw successes and failures when the dataset is too small for a useful reliability estimate.
Turn agent testing into a release decision
Use fast deterministic checks during development, a representative regression suite before release, and broader or more expensive tests at a cadence matched to change and risk. Not every small team needs an elaborate evaluation platform on its first day.
A release record should identify the candidate version, baseline version, changed components, affected task categories, test results, unresolved issues and approving owner. Run relevant tests again after changes to tools, retrieved data, permissions or prompts, not only model upgrades.
Define what blocks release. An observed unauthorised write should not be offset by a high average answer-quality score. For softer criteria, set task-specific thresholds and an agreed exception process. A passing suite is evidence about the tested cases, not a safety guarantee.
Versioned datasets and experiment comparisons can help maintain this discipline; see LangSmith’s evaluation documentation. The architecture still needs an owner for regression testing, rollback and ongoing observation. Our MLOps partner guide covers that broader operating responsibility.
Keep the scope practical
A release suite and production monitoring serve different purposes. The suite checks known requirements before deployment. Monitoring detects how the system behaves with actual traffic, unfamiliar inputs and changed dependencies. Feed investigated incidents back into regression cases, while keeping sensitive data out of test fixtures.
Powercode Group’s approach is to start with the workflow, acceptance criteria and a useful baseline before selecting tools. If your team already has a reliable suite and the skills to maintain it, another platform may add little. If outcomes are still undefined, buying a tool will not define them.
Need help turning an agent demo into a testable release? Share the workflow and the failures you need to prevent. We can discuss a bounded evaluation and integration scope.
Frequently asked questions
What is the difference between AI evaluation and agent evaluation?
AI evaluation can assess a single model response or component. Agent evaluation also checks multi-step execution, tool use, permissions and final system state. Both component and end-to-end tests can be useful.
Can an evaluation agent replace human reviewers?
Not completely. Model-based judges can scale selected checks, but require calibration and review of mistakes. Keep humans involved where ambiguity, domain knowledge or consequences require judgment.
Are deterministic tests enough?
They are strong for directly checkable requirements, but do not cover every quality criterion. A well-formed response may still cite the wrong source or solve the wrong problem.
How many examples do we need?
Enough to cover the important task categories and support the intended decision. Start with a useful baseline, then expand. Do not treat a small passing dataset as evidence that rare failures have been eliminated.
Is an evaluation function in artificial intelligence the same as an agent evaluator?
Not necessarily. The phrase can refer to a scoring function used inside a search or decision algorithm. Here, an evaluator checks an agent’s output or behavior against acceptance criteria; it need not control the agent’s own decision process.
