The dangerous thing about a bad agent trace is how convincing it is. You open one run, follow the model calls and tool invocations, and eventually find something obviously wrong. The agent called the same tool three times. It kept sending a bad argument. It spent thirty seconds wandering through a task that should have taken five. Now you have something to fix. Except you still do not know whether you have found an important problem. Maybe it happened twice in six months. Maybe it happens in one out of every five runs. The trace looks equally broken in both cases, but the engineering priority is completely different. This is where debugging agents changes once they are running at scale. A trace can tell you exactly what happened in one execution. It cannot tell you how common that behaviour is across the system. The natural response today is often to involve another model. Give it more traces. Summarize them. Use a cheaper model if there are too many. But once you know what you are looking for, many of the next questions are fairly ordinary ones. How often did this happen? Which tool is involved? When did it start? Is it concentrated in a particular kind of request? You do not need an LLM to discover those answers by reading thousands of executions. Count the failures, group them, find a few representative examples, and then bring the model back when there is something that actually requires interpretation. There are limits to this. An agent can execute cleanly and still give a terrible answer. If there is no signal for that failure, counting will not reveal it. You still need evaluation, and sometimes you still need to sit down and read the trace carefully. I started thinking about this after reading @Cometml's write-up on how they approached diagnostics for production agents. They went through several ways of having models inspect larger numbers of traces before moving much of the large-scale checking into ordinary queries over the trace data. The useful lesson was simpler than the implementation: once you know what you are looking for, you do not need an LLM to count how often it happens. I will put the blog in comments.