# Engineering example: a retry after a lost response

Run a transaction example that commits once, loses the response and recovers the result. Inspect duplicate, conflict and tenant-scope behavior.

Author: TeqEngine
Published: 2026-09-11
Updated: 2026-09-12
Canonical: https://teqengine.ai/insights/idempotent-tool-actions-example

## The decision

This synthetic SQLite example demonstrates one recorded mutation per tenant and operation key within one local database. A response can be lost after commit and a retry can recover the result without applying the mutation again. The code, tests and captured output are available to inspect and run.

- Six behavioral tests exercise the packaged source.
- Concurrent duplicates and conflicting payloads are tested.
- The example does not claim globally exactly-once external effects.

## The failure being tested

The caller requests a five-unit credit to a synthetic account. The service writes the account change and the operation result in one database transaction. It then intentionally raises a response-loss exception after commit. The caller cannot tell from that exception alone whether the business operation happened.

A retry uses the same tenant, operation key and payload. The service finds the recorded result and returns it. Reusing the key with an eight-unit payload produces a conflict. A different tenant can use the same key for a separate operation because the uniqueness boundary includes the tenant.

This illustrates an application of idempotent API design: repeated requests for the same intended action need an authoritative duplicate-handling contract. It does not rely on the model remembering that it already sent a request.

Sources: [AWS Builders’ Library: Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/).

## Run the exact published source

[Download the runnable source archive](https://teqengine.ai/evidence/idempotency-v2/source.zip). Python standard library only. Includes source, behavioral tests, recorded results and an MIT license.

[Inspect the implementation](https://teqengine.ai/evidence/idempotency-v2/example.py). Read the exact Python source included in the archive.

[Inspect the observed results](https://teqengine.ai/evidence/idempotency-v2/observed-results.json). Deterministic output captured from the packaged source.

[Read the MIT license](https://teqengine.ai/evidence/idempotency-v2/license.txt). Reuse terms for this synthetic example. The archive includes the same license.

[Verify the archive checksum](https://teqengine.ai/evidence/idempotency-v2/source.zip.sha256). SHA-256 of the published source archive.

**From the extracted idempotency-example directory**

```sh
python3 -B -m unittest -v
python3 -B example.py
```

Use Python 3.10 or later. The example uses only the standard library and creates a temporary local database. It makes no network request, calls no model and requires no credentials. The JSON output can be compared with observed-results.json in the archive.

Version 2 closes every database connection explicitly and adds a separate writer process to the recovery test. That process commits, encounters the simulated response loss and exits without returning a result. This tests recovery after a local commit; it does not simulate a network partition or a crash during storage commit. The original versioned download remains available for reproducibility.

## The important implementation choices

- A composite primary key scopes each operation to its tenant.
- The stored canonical payload distinguishes a duplicate from a conflicting reuse of the key.
- BEGIN IMMEDIATE gives the local transaction a write reservation before checking and mutating.
- The account mutation and operation result commit in the same database transaction.
- A new connection can recover the result after response loss.
- The deliberate response-loss exception occurs after commit, which exercises the ambiguous-write case.

A plain check-then-write sequence would be insufficient under concurrency. Two callers could both observe no operation record and both apply the credit. The test suite sends sixteen duplicate requests through separate connections and verifies a single recorded mutation.

## Captured result from this source

**Observed scenario output**

```json
{
  "after_distinct_tenant": {
    "alpha_balance": 5,
    "beta_balance": 7,
    "operations": 2
  },
  "after_retry": {
    "alpha_balance": 5,
    "beta_balance": 0,
    "operations": 1
  },
  "conflicting_payload_rejected": true,
  "fixture": "synthetic local SQLite transaction",
  "recovered_result": {
    "account": "account-1",
    "balance": 5,
    "version": 1
  }
}
```

After the lost response and retry, the alpha account balance is five and there is one operation record. The conflicting payload is rejected. After the separate beta operation, there are two operation records: alpha remains five and beta is seven. These are fixture outcomes, not live traffic or model benchmarks.

## What the six tests establish

**Behavioral tests**

| Test | Observation required |
| --- | --- |
| Writer exits after commit | A separate process commits and exits without returning the result. A retry recovers the original five-unit result and version 1. |
| Sixteen concurrent duplicates | One operation record and one five-unit mutation. |
| Conflicting payload | The second payload is rejected and the balance is unchanged. |
| Tenant scope | The same key can identify separate operations in alpha and beta. |
| Rejected account | No operation or mutation is left behind. |
| New intent | A new key produces a distinct legitimate mutation. |

The packaging check runs these tests from the archive bytes in a fresh directory and compares the scenario output with the published JSON. That makes the download itself part of verification, rather than assuming it matches the repository.

## The boundary of the demonstration

Caller and tenant values are trusted synthetic fixtures. This example does not implement authentication or a full authorization service. A live service must validate its caller, resource policy and domain constraints. It also needs a retention and expiry contract for operation keys.

The transaction covers one SQLite database. An external payment, email or another service requires its own duplicate handling or reconciliation. A local outbox can record intent durably but cannot, by itself, guarantee exactly-once external effects. Storage durability, isolation and failure behavior must be reviewed for the actual live system.

[Read the full retry design guide](https://teqengine.ai/insights/idempotent-ai-tool-actions). Extend the operation-identity contract to retention, unknown outcomes and external services.

## Sources and scope

Technical references inform the cited statements. The decision frameworks and synthetic examples are TeqEngine’s editorial guidance.

- [AWS Builders’ Library: Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

## Continue reading

- [Human approval workflows](https://teqengine.ai/insights/human-approval-ai-agents)
- [Reference architecture](https://teqengine.ai/insights/ai-agent-architecture)