Engineering example: an evaluation release gate

The question the gate answers

A release report can show a high pass rate while hiding one unacceptable action. The application should apply explicit release conditions to trusted evaluation evidence. A summary score is one input to that decision, not permission to ignore a known critical failure.

Agent evaluation needs a clear relationship between task outcomes, execution evidence and grading. The gate in this example starts after those outcomes have been produced; it does not validate the graders or generate the task evidence itself.[1]

The fixture contains 100 required case IDs. Every candidate must contain exactly those IDs once. A missing or duplicate case is a malformed report, not a smaller denominator that makes the pass rate look better.

Run the published decision logic

python3 -B -m unittest -v
python3 -B example.py

Use Python 3.10 or later. The example uses only the standard library, makes no network requests and calls no model. The observed-results.json file records the deterministic output from the packaged source.

The configured release policy

The critical flag is independent of the task’s pass/fail label. A case can produce the requested output and still perform a prohibited intermediate action. The gate must block that candidate even if the task is labeled pass.

Set the threshold per task. A real product needs task-specific thresholds and consequence-aware conditions, and some tasks require substantially different evidence or review.

Captured result from this source

{
  "clean_candidate": {
    "critical_cases": [],
    "decision": "pass",
    "pass_rate": 1.0,
    "passed": 100,
    "reasons": [],
    "required": 100,
    "ungraded_cases": []
  },
  "fixture": "synthetic deterministic outcomes; not model performance",
  "high_average_with_critical_failure": {
    "critical_cases": [
      "case-099"
    ],
    "decision": "block",
    "pass_rate": 0.99,
    "passed": 99,
    "reasons": [
      "critical_violation"
    ],
    "required": 100,
    "ungraded_cases": []
  },
  "minimum_pass_rate": 0.95
}

The first candidate passes 99 of 100 tasks but records a critical violation on case-099. Its decision is block, with critical_violation as the reason. The clean candidate passes all 100 tasks and the configured gate. Neither result describes actual model performance.

What the six tests establish

• A 99% pass rate cannot override a critical failure.
• A critical flag blocks even when the task itself is marked pass.
• An ungraded case is not counted as an acceptable completed assessment.
• Missing or duplicate cases are rejected.
• The ordinary pass-rate threshold is enforced independently.
• A clean candidate passes the configured policy.

The packaging process reruns the tests from the archive in a fresh directory and checks the scenario output. A reader can change a fixture, rerun the gate and inspect which reason changes. This makes the release policy reviewable as code rather than only as prose.

What passing the example does not prove

The gate trusts its input outcomes and critical flags. A faulty grader, incomplete case inventory or unobserved unauthorized action can produce misleading inputs. The live evaluation system needs representative cases, calibrated grading and reliable execution evidence before this decision logic can be useful.

Rare-failure rates, real-world input coverage and attack resistance are measured separately. The gate's job is to apply its stated policy correctly, every time.

The valuable property is inspectability: the required cases, conditions and reasons are explicit. A buyer can ask whether the proposed release process offers the same clarity when results are incomplete or uncomfortable.