AI Agent Observability: Monitoring What the Agent Decided

Your agent is in production and you cannot see it

Application monitoring answers whether the service is up and how fast it responded. Both of those can be green while an agent returns wrong numbers to a VP every morning for six weeks.

Agent observability is the practice of monitoring what the agent decided, not whether it responded. It is a different set of signals, and almost none of them come out of your existing APM tool.

The gap is not theoretical. The most common way an AI deployment fails in retail is silent degradation: a source table changes, the agent starts joining differently, the numbers shift 12%, and nothing anywhere throws an error.

Four layers of agent telemetry

Infrastructure. Latency, error rate, token spend, provider availability. Standard, necessary, and the least interesting.

Trace. The full step sequence for one task: what the agent was asked, what it retrieved, every query it wrote, what came back, what it concluded. One row per invocation is not enough. You need the tree.

Output quality. Whether the answer was right. This requires ground truth, which is why most teams skip it, and it is the only layer that measures the thing you bought.

Outcome. Whether anyone acted, and whether the metric moved. A perfectly accurate agent that changes zero decisions has a value of zero, and only this layer catches it.

Teams instrument layer 1, sometimes layer 2, and then wonder why they cannot tell whether the deployment is working.

The nine signals worth alerting on

Query drift. The same recurring question now produces different SQL than it did last month. Almost always a schema change upstream, occasionally a model version change.

Result drift. Same question, same SQL, materially different number. Either the data changed or something upstream broke.

Refusal rate. How often the agent declined to answer. A rising refusal rate is usually good news about the agent and bad news about the data.

Step count per task. A task that used to take 4 steps and now takes 30 is looping. Cap it, then alert on it.

Tool error rate by tool. One integration failing silently is the most common cause of quietly incomplete answers.

Token and warehouse cost per task, tracked together. Warehouse spend usually moves first and further.

Confidence and citation coverage. The share of answers that shipped with the query attached. Anything below 100% is a hole in your audit trail.

Human override rate. For tier 2 agents, the share of proposals a human rejected. This is your single best quality proxy and it costs nothing to collect.

Case closure rate. The share of findings where somebody acted and the metric recovered.

Ground truth without a labeling project

The standard advice is to build an eval set, and the standard outcome is that nobody does, because it sounds like a quarter of work.

The cheap version takes a day. Pull 30 questions your team answered last quarter where an analyst produced a checked number. Ten simple aggregations, ten needing a non-obvious join, ten depending on business logic that is not in the schema. Store the question and the correct answer.

Run that set weekly against production. Track three numbers: exact-match accuracy, silent failure rate among queries that executed cleanly, and refusal rate. The silent failure rate is the one that predicts whether anyone will trust the system in six months.

Re-run the set after every model version change, every schema migration, and every prompt change. Those three events cause nearly all regressions, and all three are things you already know about.

The four ways agents degrade in production

Schema drift. A column is renamed, a table is deprecated, a new column is added that is semantically close to an old one. The agent picks the new one. Nothing errors.

Model version change. Your provider ships a new default. Behavior shifts. Pin your model version and treat upgrades as changes that require an eval run, exactly like a library upgrade.

Data quality change upstream. A store stops sending a feed, a category gets remapped, a promo calendar loads late. The agent answers confidently from partial data. Row-count checks catch most of this and almost nobody runs them.

Scope creep. The agent was built for 40 questions and is now being asked 400. Accuracy on the original 40 is unchanged and overall satisfaction is falling. This one looks like a model problem and is a product problem.

Why this lands on the CIO

Nobody else is set up for it. The business owner consumes output and has no way to know the join path changed. The vendor sees their side of the call and not your schema migration. The data team owns the tables and does not see the agent.

Agent observability sits exactly where IT operations already sits: watching a running system across vendor boundaries and catching the failure before the business does. Treat it as production monitoring for a new workload type, staffed the same way.

The practical ask is small. One dashboard, nine signals, a weekly eval run, and an owner. The alternative is finding out from the CFO.

Five questions for any agent vendor

• Can I export the full trace, including every query the agent wrote, to my own store?
• Is the model version pinned, and do I get notice before it changes?
• What happens to accuracy when a source table changes, and how would I find out?
• Can I run my own eval set against production on a schedule?
• What is your override rate across your customer base, and how do you measure it?

A vendor who cannot answer the fifth question is not measuring quality at all. That is worth knowing before the contract, not after.

How Ward instruments itself

Every Ward finding ships with the query text, the row count, and the metric definition it used. Traces are exportable. Model versions are pinned and changes are announced. Case closure rate is a number we report to you rather than a number we keep.

Observability is the product, so we hold ourselves to the same instrumentation we monitor your stores with.