Interactive agents
Assistants and product features where a person waits for the answer. Speed and quality are judged at the moment of use.
Harness engineering
The Harness Method.
Build, measure and run AI agents
that stay reliable through every model change.
Harness engineering is everything around the model: context, tools, permissions, checks and recovery. The Harness Method is how we engineer it on every engagement, so your team can inspect it, own it and change it.
The Harness Scorecard
- QualityThe task done correctly, counting every attempt.
- SpeedTime to a useful result, measured where the user waits.
- CostTotal cost per accepted task, including people's time.
- ControlWhat the agent may do, and proof it stayed inside those limits.
- DriftWhat changed since the last accepted version, including after a model update.
Six questions
Six questions before more autonomy.
Every workflow starts with the same six questions. They apply to a runtime action and to a code change; the controls and the consequences differ.
- Outcome
- What useful result must the work deliver, and what would make us reject it?
- Context
- Which source, constraints and current state does the agent need?
- Authority
- What may it read, change or execute? Who can approve the next commitment?
- Evidence
- Which observations demonstrate behavior, acceptable design and the permitted path?
- Budget
- What limits apply to attempts, compute, elapsed time and human review?
- Recovery
- How do we stop, resume or reconcile work that ends partway through?
Five measures
Quality is one of five.
Most slow or wrong answers come from somewhere a single eval never looks. Every engagement scores the agent on all five, on your own cases.
Quality
The task done correctly, counting every attempt.
Speed
Time to a useful result, measured where the user waits.
Cost
Total cost per accepted task, including people's time.
Control
What the agent may do, and proof it stayed inside those limits.
Drift
What changed since the last accepted version, including after a model update.
The loop
Define. Observe. Challenge. Improve. Operate.
Each step ends with something your team can inspect: a signed Spec, a baseline Scorecard, a challenged check, a recorded decision and a live gate.
Define
Agree the outcome, the cases that decide it and what counts as an attempt.
Observe
Trace a real task end to end and record the baseline scorecard.
Challenge
Test the checks themselves, and try the simpler option.
Improve
Make the change, compare before and after, record the decision.
Operate
Keep the gate in place, record the accepted configuration and review each model release.
The system we measure
Six layers.
Two planes across them.
We trace one real task from the interface to the model and back. The time and money hide between layers, in the waits and retries no single tool reports.
State and memory, and human oversight, run through every layer. They decide what the agent knows and who can stop it.
The system we measure
- InterfaceWhere people and other systems meet the agent.
- Services & dataThe APIs, records and retrieval the work depends on.
- Orchestration & sessionPlans, steps, retries and long-running state.
- Tools & environmentThe actions the agent may take, and where they run.
- Models & routingWhich model handles which step, and why.
- Configuration & policyPrompts, settings, limits and permissions, versioned.
- State & memoryWhat the agent knows, keeps and forgets across tasks.
- OversightWho approves, who watches and who can take over.
How much to delegate
Authority grows with the evidence.
Each workflow gets an autonomy level and the evidence needed to move up one. A quarterly review decides whether the agent's authority expands, holds or contracts.
- ExpandThe evidence supports more authority.
- HoldThe agent stays at its current level.
- ContractAuthority narrows until the evidence returns.
Five rules
The rules every engagement follows.
Challenge the checker.
A green test is a claim about the test.
Count every attempt.
Fix the denominator before reading the result.
Judge the path and the end state.
A good outcome can hide a bad path.
Prove the next change.
The work is finished when your team ships the next model change through the gate.
Re-earn every component.
On every new model, keep only what still pays for itself.
Three profiles
The same method for every kind of agent.
Long-running agents
Work that runs for minutes or hours, survives interruptions and reports back. Recovery and control carry more weight.
Agents that build your software
Coding agents working inside your repositories. The result is a change your team can review, accept and maintain.
When agents build your software
The construction loop.
Agents implement bounded tasks. Engineers own the architecture, the review and the release. Every merged change is one your team understands.
Inspect an agent-built codebaseThe software construction loop
01 / A bounded task
Define the outcome, acceptance criteria and exclusions. Prepare the repository context and allowed tools.
Worked example · Synthetic data
A working export. An inspectable decision.
The task sounds simple: export customers to CSV. The contract is more specific. Only an authorized administrator may export active customers from their own tenant, with exact fields and a bounded output. An error must never become a partial download.
We made those rules explicit, implemented the change through the harness and checked the result against every one of them. The handoff is the contract, the code and the evidence together.
Includes the source, test fixtures, 24 tests and a one-line verification command that runs on any laptop.
The verification record
The right rows. Within the right boundary.
- 01Authorize
Session, role and permission before the read.
- 02Select & bound
Active rows from one tenant. Exact fields and output limits.
- 03Verify & decide
Inspect the output and every rejected case.
24 tests passed6 existing-list + 18 export tests
From a plausible result to a checked result.
- Define the expected behavior.
An administrator in tenant A can export active customers in tenant A. The output must match the contract exactly.
- Plant a failure.
Remove the tenant filter in a separate control. The CSV still looks valid, but now includes tenant B’s customer.
- Run the rejecting check.
The exact-membership check expects only
a-101. The extrab-201row causes it to fail. - Restore the boundary.
The tenant-scoped implementation returns the expected row. The full set of 24 local tests passes.
- Hand over the decision.
Keep the contract, runnable example and verification record together. The sample review identifies the identity integration, load and recovery checks needed next.
Planted failure · Check rejects
- a-101
- Tenant A · expected
- b-201
- Tenant B · outside the boundary
Corrected behavior · Check passes
- a-101
- Tenant A · exact membership
HTTP 200 · 64 bytes · exact CSV fields
The check inspects which rows came back, as well as whether the request succeeded.
The contract and observed output
- Request
GET /customers/export- Authority
- Authenticated fixture session, administrator role and export permission, checked before reading customers.
- Membership
- Active customers belonging to the authenticated tenant, ordered by customer_id.
- Bounds
- 1,000 data rows and 1,048,576 UTF-8 bytes, including the header. Exceeding either refuses the whole download.
customer_id,name,email
a-101,"North, Branch",north@example.test
The CSV preserves accepted field values, including quoted commas and embedded lines, and rejects formula-like prefixes that could execute in a spreadsheet.
Verification and counterexample
Check what must not happen.
We planted a deliberately unscoped version that returned another tenant’s row. The exact-membership check caught it; restoring the tenant-scoped implementation passed.
- Session, administrator and export permission required before customer read.
- Exact membership and fields, second tenant, active-only selection.
- Client headers cannot change identity or permission; all query parameters refused.
- Quotes, commas, embedded lines and ordinary leading spaces round-trip.
- Scoped formula-prefix policy applies to every exported field.
- Empty export includes header; export is not just the first list page.
- Exact row limit and encoded-byte limit, including CSV header.
- Database failure and exceeded bounds produce no partial download.
- Only GET supported; existing list behavior preserved.
A second example: this website’s contact API
Check the effect, not just the response.
A valid, verified inquiry should reach the email function once. Failed verification must not call it. An email-service error must not return a false success response.
The handler tests check the HTTP result and the email call together, so a green response always means the message was handed off.
What your team inherits
The software, and the means to keep changing it.
The handoff is planned before implementation starts. The work is finished when your team ships the next model change through the gate, or keeps us on monthly to run it.
Repository context
Project guidance, architecture decisions, task conventions and the commands needed to build and check the software.
Verification & review
Project-specific tests, acceptance gates and review packets tying the change to its evidence and remaining risks.
Operation & recovery
Agreed workflows, configuration, runbooks and the procedure for handling an interrupted or failed run.
Ownership & access
Named maintainers, client-approved accounts and a rehearsed handoff, or a monthly retainer if you want us to keep running it.
Where to start
Put the method to work.
Delegate the implementation. Engineer the judgment. Own the result.
Build together.
Forward-deployed engineers build the agent with your team, measured with the method from the first week.
Forward-deployed buildReview what runs.
Map an agent across every layer, score it on the five measures and rank what to fix first.
The Harness ReviewWatch every release.
Rerun your scorecard on each relevant model release and decide adopt, hold or switch on the evidence.
Harness WatchDesign the first one.
Choose the workflow worth automating, write the Harness Spec and prove a scored first version.
Harness DesignStart a conversation
What are you building—or deciding?
Tell us what you’re building, what is getting in the way, and the outcome you need.