Ask a leadership team what their agent did yesterday and the honest answer is usually a count. It ran 1,400 times. Nobody can say how many of those worked, what they cost, or why run 4812 sent a customer a refund twice. This essay is about the observability that the Scale stage of Discover, Design, Build, Scale depends on, written for the people who approve the budget rather than the people who wire the tracing.
Observability is not a dashboard
A dashboard shows totals. Observability is the ability to pick any single run and answer, in minutes, four questions: what did it do, what did it cost, did it work, and who is responsible. If any of the four needs an engineer to grep logs for an afternoon, the agent is not observable, however good the dashboard looks.
The reason this matters at Scale and not before: with one agent and a small team, people hold the system in their heads. At Scale the people change, the models change, and the second agent starts calling the first. You cannot debug what you cannot trace, and you cannot scale what you cannot debug.
The four things, per run
What it did. A trace: the tree of steps the agent took, each model call and tool call as a node, with the inputs and outputs attached. Not an event log, which tells you something happened, but a tree, which tells you why. When agent A hands off to agent B, the trace crosses that boundary without losing the thread. In practice this means every span carries a run identifier, a version stamp of the agent and prompt that produced it, and the identity it acted under.
What it cost. Tokens in and out, per model call, rolled up to a cost per run, in currency. Not a monthly bill divided by a run count, which hides the one workflow that costs ten times the others. Cost on the trace is what lets an owner notice that a prompt grew or a retry loop appeared, the week it happened.
Whether it worked. The verdict of a check that is not the model: tests passed, ticket closed without reopen, person approved. Recorded on the run, so pass rate is a query, not a survey. This is the same check that gates the eval and judges the holdout. If the run was in the holdout, that is recorded too, so the benefit can be computed from the traces themselves.
Who owns it. The agent, its version, and the named owner, on every run. When something goes wrong at 3 a.m., the page goes to a person, and that person can see the trace that caused it. The 3 a.m. runbook assumes this exists.
Why I insist on a portable format
Every cloud will sell you observability for the agents built on its platform. Azure has it for agents on Foundry; Google has it for agents on Vertex. Both are good. Both are also a lock-in tax if they are the only place your traces can live, because the moment you have an agent on the other cloud, or a tool that runs on neither, you have two half-pictures.
So I instrument once, in OpenTelemetry's conventions for generative AI, and send the traces wherever the client already looks. The field guides for Azure AI Foundry and Vertex AI show how to do that without giving up the native tooling. Google's own framework lists distributed tracing, structured logging and assigned roles for monitoring as the core of what it calls AgentOps. I agree with the list and add the portability requirement.
What this lets a leader do that a dashboard never will
- Answer the incident question. "Why did it do that" becomes a link to a trace, not a meeting.
- See cost by workflow, not by month. The expensive workflow gets fixed or retired; the cheap ones get more volume.
- Trust the benefit number. The holdout verdict is computed from the same traces that show the runs, so nobody has to take the vendor's word for it.
- Catch drift with no deploy behind it. A model update or a changed document shows up as a pass-rate drop on the trace stream before a customer notices.
- Approve the second agent with evidence. The first agent's traces are the business case for the next one.
A five-minute test for the agent you have
Pick one run from last week at random. Ask your team to tell you, within the hour, what it did step by step, what it cost in currency, whether it passed the client's own check, and who would have been paged if it had failed. Time how long it takes and how many people it involves. If the answer is one person and ten minutes, you have observability. If it is three people and tomorrow, you have a dashboard.
Running that test, and mapping the gap between what you have and the four questions, is one of the first things I do in a free AI readiness assessment. It is also the shortest path I know from "we have an agent" to "we run agents".