Engineering example: a retry after a lost response

The failure being tested

The caller requests a five-unit credit to a synthetic account. The service writes the account change and the operation result in one database transaction. It then intentionally raises a response-loss exception after commit. The caller cannot tell from that exception alone whether the business operation happened.

A retry uses the same tenant, operation key and payload. The service finds the recorded result and returns it. Reusing the key with an eight-unit payload produces a conflict. A different tenant can use the same key for a separate operation because the uniqueness boundary includes the tenant.

This illustrates an application of idempotent API design: repeated requests for the same intended action need an authoritative duplicate-handling contract. It does not rely on the model remembering that it already sent a request.[1]

Run the exact published source

python3 -B -m unittest -v
python3 -B example.py

Use Python 3.10 or later. The example uses only the standard library and creates a temporary local database. It makes no network request, calls no model and requires no credentials. The JSON output can be compared with observed-results.json in the archive.

Version 2 closes every database connection explicitly and adds a separate writer process to the recovery test. That process commits, encounters the simulated response loss and exits without returning a result. This tests recovery after a local commit; it does not simulate a network partition or a crash during storage commit. The original versioned download remains available for reproducibility.

The important implementation choices

• A composite primary key scopes each operation to its tenant.
• The stored canonical payload distinguishes a duplicate from a conflicting reuse of the key.
• BEGIN IMMEDIATE gives the local transaction a write reservation before checking and mutating.
• The account mutation and operation result commit in the same database transaction.
• A new connection can recover the result after response loss.
• The deliberate response-loss exception occurs after commit, which exercises the ambiguous-write case.

A plain check-then-write sequence would be insufficient under concurrency. Two callers could both observe no operation record and both apply the credit. The test suite sends sixteen duplicate requests through separate connections and verifies a single recorded mutation.

Captured result from this source

{
  "after_distinct_tenant": {
    "alpha_balance": 5,
    "beta_balance": 7,
    "operations": 2
  },
  "after_retry": {
    "alpha_balance": 5,
    "beta_balance": 0,
    "operations": 1
  },
  "conflicting_payload_rejected": true,
  "fixture": "synthetic local SQLite transaction",
  "recovered_result": {
    "account": "account-1",
    "balance": 5,
    "version": 1
  }
}

After the lost response and retry, the alpha account balance is five and there is one operation record. The conflicting payload is rejected. After the separate beta operation, there are two operation records: alpha remains five and beta is seven. These are fixture outcomes, not live traffic or model benchmarks.

What the six tests establish

The packaging check runs these tests from the archive bytes in a fresh directory and compares the scenario output with the published JSON. That makes the download itself part of verification, rather than assuming it matches the repository.

The boundary of the demonstration

Caller and tenant values are trusted synthetic fixtures. This example does not implement authentication or a full authorization service. A live service must validate its caller, resource policy and domain constraints. It also needs a retention and expiry contract for operation keys.

The transaction covers one SQLite database. An external payment, email or another service requires its own duplicate handling or reconciliation. A local outbox can record intent durably but cannot, by itself, guarantee exactly-once external effects. Storage durability, isolation and failure behavior must be reviewed for the actual live system.