AI agent observability: from traces to decisions

Start with the operating questions

A dashboard is useful when it supports a decision. Can users complete the intended task? Are unauthorized actions being attempted or blocked? Which failure class is increasing? Is review effort growing? Are timeouts producing unknown writes? These questions determine what the system should record.

Traditional service signals such as latency, traffic, errors and saturation remain useful. Google’s SRE guidance explains these core monitoring concerns. An agent product adds task-level outcomes and model/tool behavior; it should not discard the service layer that makes those outcomes possible.[1]

Design the trace around stable identities

Use a task ID to connect the product interaction with model calls, retrieval, tools, approvals and the final outcome. Use a separate operation ID for each intended business mutation. A retry should preserve that operation identity; a new model response should not silently create a new business action.

task_id
operation_id (when a business action exists)
event_type
status + reason_code
model / prompt / tool / policy versions
duration_ms + usage counters
evidence_reference (restricted, when needed)

The exact fields depend on the product. Avoid high-cardinality or sensitive content in metric labels, including raw prompts, email addresses and document text. A trace can contain a controlled evidence reference without copying the underlying record into every telemetry backend.

OpenTelemetry maintains GenAI conventions for spans, metrics and events in a dedicated repository. Pin the conventions and instrumentation versions that an implementation uses; do not assume every provider integration emits the same fields or that old documentation paths remain current.[2]

Worked investigation: the model is not the bottleneck

A synthetic support workflow takes 42 seconds from submission to useful completion. Its trace shows 2 seconds of queue wait, 3 seconds of model activity, 4 seconds of tool activity, 31 seconds waiting for approval and 2 seconds for commit and confirmation. These are illustrative intervals on one serial path, not measured live results.

Switching to a faster model cannot remove the 31-second approval wait. The team should inspect the review interface, routing and whether the approval is required for this action class. It should not remove a necessary control merely to improve the latency chart.

Keep both views. Excluding approval time from every chart makes the product look faster than the experience. Including it in model latency sends engineers toward the wrong optimization.

Use graduated evidence access

Most operating questions need states, timings, versions and reason codes. Some investigations need the exact permitted input or output. Make that access a deliberate layer with retention, redaction and authorization appropriate to the data. Do not assume that a telemetry provider should receive everything the application can read.

• Default records: task status, component timing, bounded usage, version identifiers and reason codes.
• Restricted evidence: selected input/output artifacts needed to reproduce a failure, with a defined retention period.
• Escalated investigation: access granted for a specific operational purpose, with a record of who accessed the evidence.
• Excluded by default: credentials, unnecessary personal data and hidden model reasoning.

Record observable actions and outputs rather than claiming to reconstruct the model’s internal decision process. An explanation generated by the model can be useful context, but it is not an authoritative audit of why the computation behaved as it did.

Alert on conditions an operator can act on

Define thresholds from the workflow’s requirements and observed baseline. A spike in blocked malformed calls may deserve investigation; a single blocked adversarial request may be expected behavior. A growing queue of unknown financial operations may warrant immediate intervention even if the overall task-success average remains high.

An alert should link to the relevant task cohort, versions and runbook. A red chart without an owner or a safe intervention is only a notification.

Close the loop with evaluation

Turn recurring live failures into reviewed evaluation cases using data the team is permitted to retain. Preserve the failure class and expected behavior, not merely a copied transcript. Compare later releases against those cases and representative held-out work.

This creates a useful loop: observe an outcome, inspect the boundary, reproduce the failure and verify a correction. It gives the engineering team evidence for a release decision and gives a buyer a concrete way to judge whether the system will be maintainable.