🔀 Dynamic Model Escalation For Efficient Evals 💰 Most eval pipelines pick one judge model and run it on everything. If it’s a big model, it’s too slow and too expensive to score production traffic, so teams sample a few percent and hope. If it’s a small model, it’s cheap enough to score every trace, but nobody knows which of its scores to trust. The question isn’t which judge model to use. It’s when the judge should admit it doesn’t know. Here is what we have built at @fiddler_ai as part of our Centor Evaluation Runtime. Two models function with two distinct jobs. A decisioning model sits in the hot path and scores every span/trace in under 100ms. A reasoning model sits behind it and only ever sees what the decisioning model sends it. The interesting part is the handoff. The decisioning model doesn’t just emit a verdict. It emits a verdict plus an uncertainty score about how much that verdict should be trusted, derived from the trace itself and from where this verdict sits relative to everything the model has scored before. That signal is what routes. Traces the decisioning model is sure about keep their score, and that’s the majority of traffic. Traces where the signal says the model is operating outside its competence go to the reasoning model, which re-scores them with the time and context a fast model can’t afford. So the split isn’t fast versus accurate. It acutally works as decide JIT versus deliberate post-hoc analysis. The decisioning model’s job is to be right quickly and to know when it isn’t. The reasoning model’s job is to be right on the cases that need thought. One eval score comes out the other end, and the user never needs to know which model produced it. Ofcourse we keep an audit of all of this if user wants to actually know the details. This is very similar to how humans perform a good on-call rotation. The pager doesn’t wake the senior most engineer (large reasoning model) for every alert. It wakes them for the ones that the on-call engineer (fast decision model) using the the runbook can’t resolve. So, what are the main advantages of this: The reasoning model’s bill is a function of how unusual your traffic is, not how much of it there is, which is the only way scoring every production trace stays affordable. And the escalation queue becomes the most useful dataset you have: a running list of where the decisioning model is weak and where your traffic is drifting. When you label those, feed them back into the next version of the decisioning model, and the escalation rate drops over time. The eval system tunes itself on its own misses. Welcome to the world of self-tuning AI Observability! Eval Models: fiddler.ai/centor-models

As AI writes more code, fewer people will understand in detail how that code behaves in production. That makes observability more important, and it changes what we should expect from it. Infrastructure teams are building agents to investigate incidents. AI teams are building tools to trace and evaluate agents. These efforts are starting to meet: agents need production data to do their work, and we need that same data to verify their work. Take a customer support agent processing a refund. Every service responds successfully. Database latency is normal. The model gives a coherent answer. The refund API works exactly as designed. But the agent refunds the wrong order. Business logic failures aren’t new. What changes with agents is how much of that logic gets decided at runtime, based on prompts, retrieved data, memory and the results of earlier actions. To investigate, you need to connect those steps. Which order did the agent retrieve? What information did it have? Which tool did it call, with what arguments? What actually changed in the transaction system? “I completed the refund” is a claim we need to check against the transaction record. A study of nearly 9,900 tau2-bench runs examined this problem: agents declaring success when the environment showed otherwise. The frequency varied substantially across domains. In the benchmark’s single-control domains, these false success claims accounted for 45–48% of failures. That makes task completion an essential part of observability. A task can span multiple requests, models, tools and human approvals. We need the execution details to understand what happened, and evidence of the outcome to establish whether it worked. It also changes how we measure cost. A model with cheaper tokens may need more retries, more tool calls and more human intervention. The meaningful comparison is cost per successfully completed task. Then there’s the question of who uses the telemetry. An engineer investigating an incident can read a dashboard, remember yesterday’s deployment and ask a colleague about an undocumented dependency. An agent needs access to that context through tools and data. A log search API helps. But the investigation also needs to connect a symptom to a code version, a configuration change, a dependency and the customer workflows affected. Without that context, an agent can produce a convincing explanation that the evidence doesn’t support. @mipsytipsy has been making a useful point: AI demands more engineering discipline. As generating code gets easier, we need stronger ways to establish what it actually does. I think this brings observability and evals much closer together. A production failure should become a test case. A proposed fix should be evaluated against that case and other representative tasks. After deployment, we should check whether the improvement holds under real traffic. The definition of success needs to carry through that process. If we evaluate one thing before release and monitor something different afterward, we lose the feedback needed to improve the system. Measuring success takes work, too. An LLM judge gives us an estimate that needs to be checked against domain expertise and actual outcomes. A high helpfulness score tells us very little about whether the right order was refunded. For that question, checking the transaction record is more useful. This is where I see the opportunity for observability products. Collecting telemetry remains essential. The value comes from helping engineers and agents use it to identify a problem, test an explanation and verify that a change fixed it. Agents will do more of the querying and investigation. Humans will still need to examine ambiguous failures and decide which outcomes matter. Both need access to the underlying evidence, especially when the automated diagnosis is wrong. We’re getting much better at generating software. We need to get equally good at understanding what it does after we ship it.