Taking an AI prototype to launch

Write the product contract before hardening the prototype

A prototype often succeeds because the person demonstrating it supplies missing context and repairs exceptions. Write down what that person is doing. Which inputs are cleaned manually? Which wrong answers are discarded? Which accounts are accessible? Which failures are repaired by retrying the demo? These are product requirements waiting to be made explicit.

Define the user, task boundary, expected volume, acceptable outcome and unacceptable actions. State what happens when the task cannot be completed. A useful unresolved result can be better than a confident answer that changes the wrong record. The contract should cover both the final output and consequential intermediate actions.

Use rollout states with distinct evidence

The table is a planning pattern, not a universal release schedule. A read-only assistant and an agent that changes financial records need different evidence. Promotion should depend on the consequence level and observed behavior of the actual workflow.

Build a useful evaluation set

Include ordinary tasks, difficult but valid tasks, missing information, unauthorized requests, contradictory evidence and failures of dependencies. Record expected outcomes at the level the product promises. If the system should create a valid proposal, testing only whether its final sentence sounds helpful misses the core requirement.

Agent evaluation benefits from separating the task outcome, execution trace and grading method. Anthropic’s evaluation guidance discusses these distinctions and the need to inspect how a task was completed. Use them to make failures diagnosable, rather than collapsing everything into one accuracy score.[1]

Keep a held-out set for release decisions. Preserve important regressions as named cases, but do not treat repeated success on a small familiar suite as proof of broad reliability. Record the model, prompt, tool schema, data snapshot and grader versions so results can be interpreted later.

Make operation a first-class deliverable

A complete build needs observable task states, scoped access, deployment controls, ownership and an incident path. Those requirements are not unique to AI. Secure software development practices still apply to source, builds, dependencies and verification.[2]

• Show whether each task is waiting, executing, complete, rejected or unresolved.
• Track accepted outcomes and review effort alongside execution cost.
• Record operational evidence with access and retention rules; do not log sensitive context indiscriminately.
• Define who can disable an action class or integration and how quickly that change takes effect.
• Make configuration, evaluation fixtures and recovery instructions part of the handoff.

An operator should be able to distinguish a slow model from a queue backlog, a waiting approval and a downstream timeout. Otherwise the first response to an incident becomes guesswork, and the proposed fix may target the wrong component.

Rehearse recovery with a concrete failure

In a synthetic account-update workflow, the write commits but the worker loses its connection before recording the response. The task remains unresolved. Recovery should query the original operation ID and record the committed result. Re-running the entire task with a new operation ID risks applying a second update.

A rollback of the AI feature might disable new tasks and new commits while allowing reconciliation of existing operations. It should specify how pending approvals are invalidated and how users see the status of work already started. A deployment rollback cannot automatically reverse every business side effect.

What should be true before expanding

The team can demonstrate the accepted task, the important denied cases, the operating budget and the recovery path. Reviewers understand the remaining limitations. The product can stop or escalate honestly when it reaches them. The release record names what changed and why the evidence supports the next stage.

If those conditions are missing, the next work is not necessarily more model tuning. It may be a smaller task boundary, a reliable domain API, a better fixture or clearer operational status. Those changes often determine whether the useful prototype becomes a product people can depend on.