# AI agent evaluation: a release framework

Define task success, test permissions and side effects, evaluate failure slices, and turn agent evaluations into an inspectable release decision.

Author: TeqEngine
Published: 2026-09-11
Updated: 2026-09-12
Canonical: https://teqengine.ai/insights/ai-agent-evaluation-framework

## The decision

Evaluate an AI agent at three levels: the business outcome, the actions it took, and the components that caused failures. A release decision should combine representative task results with hard constraints on unauthorized or unsafe behavior, plus cost and latency limits.

- Check the resulting business state, not only the final response.
- Separate release blockers from average quality scores.
- Version test data, graders and system configuration together.

## Define success before choosing an evaluation tool

An agent evaluation framework is the combination of tasks, expected outcomes, checks, execution infrastructure and decision rules used to judge a system. The tool that runs those checks is one part. Starting with a dashboard or a benchmark score can obscure whether the system completes the job your users need.

For a support workflow, success may mean resolving the right ticket with an accurate, policy-compliant response. For a scheduling workflow, it may mean a valid reservation actually exists. An agent saying it completed an action is not enough. Anthropic distinguishes the recorded execution from the resulting environment state; that distinction is especially useful for agents that use tools.

Sources: [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents).

Write an acceptance contract for each task. Describe the starting state, the allowed actions, the expected final state and the evidence used to check it. Include cases where the correct behavior is to refuse, request clarification or escalate. Completion rate alone should not reward an agent for acting when it should stop.

## Use three layers of evaluation

**What each layer can establish**

| Layer | Question | Example evidence |
| --- | --- | --- |
| Outcome | Did the business task finish correctly? | The requested record exists once, belongs to the correct customer and contains approved values. |
| Execution | Were the actions authorized and appropriate? | Tool-call records show the expected identity, approved operation and no prohibited side effects. |
| Component | What caused the failure? | Retrieval missed the current policy; a tool argument was malformed; a timeout handler retried a completed write. |

Use deterministic checks when the outcome can be computed: exact identifiers, schema validity, authorization decisions, duplicate records or required state transitions. Use expert review for ambiguous judgments. A model-based grader can help evaluate a defined rubric, but it needs calibration against human decisions and monitoring for disagreement.

Do not make a single model judge responsible for deciding whether an authorization boundary was respected. The enforcement system and resulting records should provide that evidence. Also avoid forcing every successful task through one prescribed sequence when several authorized paths are valid. Grade required invariants and outcomes, then use traces to diagnose the differences.

OpenAI’s evaluation guidance recommends task-specific evaluation, realistic data, automation where appropriate and calibration with human feedback. Those principles are compatible with many frameworks; the release decision should remain understandable outside the tool that computed it.

Sources: [OpenAI: Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices).

## Build a representative, versioned task set

Start with real workflow examples that you are permitted to use, then remove or protect sensitive information. Add synthetic cases to cover known edge conditions, but label them and do not confuse their frequency with real-world prevalence. Keep a held-out set for assessing changes that were developed against the initial examples.

- Ordinary tasks: the work the system is expected to handle most often.
- Ambiguity: missing identifiers, conflicting records, underspecified requests and stale information.
- Boundary cases: another customer’s record, revoked permission, prohibited action and untrusted instructions inside retrieved content.
- Integration failures: timeout before or after a write, rate limiting, partial response and unavailable dependencies.
- Recovery: interruption, retry, duplicate request, cancellation and a human correction.
- Distribution slices: customer types, document formats, languages, record sizes and important workflow variants.

For each case, store a stable ID, provenance, initial state, expected outcome, allowed and forbidden actions, grading rubric and risk category. Preserve the case version with every result. A changing dataset without version history can make an apparent quality improvement impossible to interpret.

> **Dataset size is a design decision**
> There is no universal number of cases that proves readiness. Size and repetition depend on variation, risk, available examples and the precision needed for the decision. A small diagnostic suite is useful; it should not be described as evidence about rare failures.

[Plan a zero-failure sample](https://teqengine.ai/insights/ai-evaluation-sample-planner). Inspect the assumptions and limits of a one-sided binomial bound before interpreting a perfect test run.

## A worked acceptance contract

This synthetic example concerns a billing-support agent that may prepare a credit recommendation but cannot issue the credit. It illustrates an evaluation structure, not a claim about a deployed system.

**Example case: duplicate billing complaint**

| Field | Acceptance condition |
| --- | --- |
| Starting state | A signed-in support user, a specified customer, two invoice records and the applicable credit policy. |
| Allowed actions | Read records within the user’s access, retrieve the policy and create a draft recommendation. |
| Required outcome | A draft linked to the correct account, with the discrepancy and supporting records identified. |
| Prohibited outcome | An issued credit, an account change, or access to another customer’s records. |
| Ambiguous input | Ask for the missing invoice identifier rather than guessing an account. |
| Unavailable dependency | Report that the necessary evidence could not be retrieved and preserve a recoverable state. |
| Checks | Inspect the draft, the tool-call audit record and the absence of a credit transaction. |

This contract lets product, engineering and the workflow owner disagree constructively. If issuing a credit later becomes part of the product, the action boundary and acceptance criteria must change. That is a new capability to evaluate, not just a more persuasive prompt.

## Turn results into release gates

Separate hard blockers from quality targets. A weighted average can hide a dangerous failure inside hundreds of routine successes. The following is an illustrative release policy; thresholds must be chosen for the real workflow and consequence of failure.

**Example release decision structure**

| Gate | Decision rule |
| --- | --- |
| Authorization and prohibited side effects | Any observed unauthorized read or write blocks the release and triggers investigation. Passing the tests does not prove none are possible. |
| Task outcome | Meet the agreed success target on representative tasks and important slices; inspect uncertainty and sample size. |
| Regression | Compare with the previous accepted version on the same versioned tasks. Explain material regressions before rollout. |
| Human workload | Review and escalation remain within the capacity the business has agreed to provide. |
| Cost and latency | Meet the specified workload budget and response-time objectives, including unsuccessful requests. |
| Recovery | Demonstrate the required stop, retry, reconciliation and rollback behavior in an appropriate test environment. |

Report both numerator and denominator. “94% success” means little without the number of tasks, definition of success, task mix, repetition and exclusions. Show the failures alongside the headline number. A model may perform well on ordinary work while failing the small slice that matters most to a regulated customer.

Repeated runs reveal variability, but they do not create new independent task coverage. Repeating the same ten examples many times is different from testing a thousand distinct workflows. Keep task coverage and repeatability visible as separate questions.

## Evaluate the evaluation system

A grader can be wrong, especially when a task admits several valid answers. Write a rubric with examples of acceptable and unacceptable behavior, then compare grader decisions with knowledgeable human reviewers. Investigate disagreements instead of averaging them away.

- Check whether the grader rewards style, length or confidence instead of task correctness.
- Keep expected answers and private grading instructions outside the system under test.
- Inspect whether an output can influence the grader through embedded instructions.
- Version the judge model, rubric and configuration; changes to them can alter scores without changing the agent.
- Use direct state checks for facts that do not require subjective judgment.
- Review false passes as well as false failures; a lenient grader can make a broken release look ready.

You need not expose hidden model reasoning to perform a useful review. Observable inputs, authorized tool calls, tool results, configuration, relevant outputs and final state can provide the operational evidence. Apply appropriate redaction and access control to traces; observability should not become an uncontrolled copy of customer data.

[Calibrate an LLM judge](https://teqengine.ai/insights/llm-as-a-judge). Use a worked confusion matrix to distinguish overall agreement from missed unacceptable answers.

## Connect offline evaluation to controlled operation

Run the relevant suite when the model, prompt, tool contract, retrieval pipeline or policy changes. Link the result to the exact system version. Preserve the configuration and evidence needed to understand a release months later.

Then use a staged rollout with monitoring appropriate to the risk. Shadow operation can observe proposed decisions without committing side effects; a limited rollout can reveal real workflow variation. Define stop conditions, escalation and recovery before starting. Neither approach replaces the need to test the action boundary.

Add newly observed failures to the test set after resolving privacy and representativeness questions. Track drift in the input mix, review effort, cost and failure categories. The aim is a repeatable loop from an observed problem to a test, a fix and a checked release.

[Evals, LLMOps and reliability](https://teqengine.ai/services/evals-and-reliability). See TeqEngine’s work on evaluation suites, retrieval diagnostics, release gates and operational evidence.

## Questions this guide answers

### What makes an AI evaluation useful for a release decision?

Define the accepted task outcome, representative cases, grading method and critical failures before interpreting a score. Record the tested configuration and inspect execution evidence as well as the final answer. The result should support a specific release decision and state which workloads and failure types remain untested.

### Is a high average pass rate enough to release an agent?

No. An average can conceal a prohibited action or access violation. Keep critical release conditions separate from aggregate task quality, and inspect failures by category. A configured quality threshold is a policy choice supported by task evidence; it does not make every failure within the remaining percentage acceptable.

### How should evaluation cases change as the system evolves?

Keep held-out cases for assessing changes, and turn permitted real-world failures into reviewed regression cases. Record expected behavior, software versions and the reason each case matters. Add cases deliberately without treating a synthetic edge case's frequency as an estimate of how often that failure occurs among users.

## Sources and scope

Technical references inform the cited statements. The decision frameworks and synthetic examples are TeqEngine’s editorial guidance.

- [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
- [OpenAI: Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices)

## Continue reading

- [Inspect MCP security](https://teqengine.ai/insights/mcp-server-security)
- [AI launch readiness checklist](https://teqengine.ai/resources/ai-launch-readiness-checklist)
- [Model cost per successful task](https://teqengine.ai/insights/ai-agent-development-cost)