# AI evaluation sample planner

Calculate a one-sided zero-failure binomial bound and the trials needed for a target. Check the statistical assumptions before using the result.

Author: TeqEngine
Published: 2026-09-11
Updated: 2026-09-12
Canonical: https://teqengine.ai/insights/ai-evaluation-sample-planner

## The decision

Zero observed failures does not establish a zero failure rate. Under a fixed independent binary-trial model, this planner calculates an upper failure-rate confidence bound and the sample size needed for a chosen target. It is useful for planning ordinary evaluation coverage; it does not certify system safety.

- At 95% confidence, a 1% target needs 299 all-success trials.
- The entered trials must have zero observed failures.
- Correlated cases, adaptive stopping and population shifts need different analysis.

## Explore the zero-failure bound

Interactive worksheet: https://teqengine.ai/insights/ai-evaluation-sample-planner#worksheet

The method and complete static example appear in this article. Inputs stay in the browser and are not submitted.

## The model comes before the number

Assume a fixed number of independent trials, each drawn representatively from a defined population, with the same underlying failure probability and an unambiguous binary pass/fail rule. The calculation below applies when all observed trials pass. NIST documents exact one-sided and two-sided binomial confidence limits; this tool uses the zero-failure one-sided case.

Sources: [NIST: Exact binomial confidence limits](https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/exacbino.htm).

An evaluation dataset does not automatically satisfy those assumptions. Several paraphrases of one task can share a failure cause. Repeating a known case under the same context can be highly correlated. Tuning the system against the evaluation set changes what later success on that set establishes.

> **Zero failures only**
> If any trial failed, do not enter only the successful trials here. Use an interval method that includes the failures and the full sample, and review any critical failures separately.

## Derive the bound from the probability of zero failures

For failure probability p and n independent trials, the probability of observing zero failures is (1 − p)^n. Set that expression equal to alpha, where alpha = 1 − confidence, and solve for the one-sided upper boundary.

**Zero-failure calculation**

```text
alpha = 1 − confidence
upper failure bound = 1 − alpha^(1 / n), for n > 0
required trials = ceil(log(alpha) / log(1 − target failure rate))
```

The implementation uses log1p and expm1 forms to preserve numerical precision for small probabilities. Trial count must be a whole number. With zero trials, there is no bound to report. The sample-size result assumes all required trials pass.

A confidence bound is a statement from a sampling procedure. It is not the probability that this particular system is safe, nor a claim that the next task has no chance of failing. The population, sampling and grading assumptions remain part of the conclusion.

## Static example: 299 trials and a 1% threshold

At 95% confidence, alpha is 0.05. For a 1% target failure rate, ceil(log(0.05) / log(0.99)) is 299. At a true 1% failure rate, the probability of 299 consecutive successes is about 4.95%, while 298 successes is just above 5%.

**Zero-failure examples at 95% one-sided confidence**

| All-success trials | Upper failure-rate bound |
| --- | --- |
| 30 | About 9.50% |
| 100 | About 2.95% |
| 300 | About 0.9936% |
| 1,000 | About 0.2991% |

These figures explain why a small perfect demo is weak evidence about infrequent failures. They also show why a fixed count such as “100 tests” is not a universal release standard. The appropriate conclusion depends on the failure rate the task needs to bound and on whether the sample is valid.

## Where the simple model breaks down

- Correlated trials: tasks share a document, dependency, user or underlying failure mode.
- Adaptive stopping: repeatedly checking the result and stopping when it looks favorable changes the procedure.
- Distribution shift: live users or inputs differ from the evaluated population.
- Ambiguous grading: the pass/fail rule is inconsistent or the grader misses failures.
- Multiple comparisons: selecting a favorable result across many models or task slices changes the inference.
- Critical events: rare unauthorized actions may require explicit controls and adversarial tests beyond a population-rate estimate.

Keep named negative cases and deterministic security checks even when a broad sample looks good. A numerical bound should not average away a known prohibited action. For heterogeneous tasks, report important slices and seek a sampling design appropriate to those differences.

## Use it as a planning conversation

Write the task population, failure definition and sampling method beside the calculated number. Decide before running the experiment how the sample is selected and what happens if failures occur. Preserve the complete results rather than reporting only successful runs.

[Inspect an evaluation release gate](https://teqengine.ai/insights/evaluation-release-gate-example). See why an explicit critical failure blocks a release independently of aggregate quality.

The planner runs in the browser and submits no inputs. The static derivation and example are available without JavaScript. A release decision still needs task-specific judgment, valid evidence and the relevant operating controls.

## Sources and scope

Technical references inform the cited statements. The decision frameworks and synthetic examples are TeqEngine’s editorial guidance.

- [NIST: Exact binomial confidence limits](https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/exacbino.htm)

## Continue reading

- [Evaluation framework](https://teqengine.ai/insights/ai-agent-evaluation-framework)
- [Judge calibration](https://teqengine.ai/insights/llm-as-a-judge)