Tracing and Observability
An AI agent can behave differently across runs on the same input. Without records, an unexpected result cannot be traced to its cause. Observability means keeping the internals inspectable after the fact.
What to Record
Record per step. A final output alone does not reveal where a problem started.
| Recorded item | Purpose |
|---|---|
| Each step’s input | What evidence was used |
| Tools called and their arguments | Whether selection was appropriate |
| Tool results | Whether the expected information came back |
| Elapsed time | Where time is going |
| Failures and retries | Whether the same failure repeated |
| Approval requests and outcomes | Whether operations that should stop did stop |
Hooks capture all of this automatically. Because it does not depend on instructing the AI, nothing goes unrecorded.
Detail Enough to Isolate a Cause
Too much detail becomes unreadable; too little cannot locate the problem. The right level is enough to isolate which step went wrong.
# Too coarse
Task run → failed
# Appropriate
Step 1 research → success (tool A, 2 results)
Step 2 generate → success
Step 3 verify → failed (second completion condition not met)Records Feed Evaluation
Accumulated records become the input to agent evaluation.
- Which steps concentrate the failures
- Which tool selections go wrong most often
- Where the step count inflates
This makes improvement targets a matter of data rather than impression.
Handle Records Carefully
Records include user input and data retrieved by tools, which may carry personal or confidential information. Decide the handling in advance.
- Define which fields are stored and for how long
- Limit who can access them
- Mask sensitive values at write time
Summary
- Agent behavior varies across runs, so causes cannot be traced without records
- Record input, tools, results, failures, and approvals per step
- Keep detail at the level that isolates which step failed
- Define retention scope and access, since records can carry sensitive data