Bench Summit by Kaggle has officially kicked-off! Trust in benchmarks is eroding. The path forward starts here. We started strong with @janiejgu sharing key insights on the problems with benchmarks today and Kaggle’s mission to build trust in evals and help the world discover what works and doesn’t work in AI. During the session, she shared four ways benchmarks fail today: 1. Relevance: Many public benchmarks don’t resemble what your agents do in production. 2. Saturation: Evals go stale within a release cycle. 3. Verification: Everyone has their own implementation of the same benchmarks, meaning scores vary wildly due to infrastructure, not model quality. 4. Infrastructure: Evals infra is hard to build correctly, and everyone rolls their own. To get numbers you can actually trust, Janie highlighted four essentials: 1. Real workflows as the foundation for benchmarks 2. Continuous benchmarks from crowdsourced tasks, arenas, and production data 3. Rigorous and transparent methodology audited by the community experts 4. Eval infrastructure that everyone can use out of the box Here’s a sneak peak of the early moments:

OpenAI, Anthropic and other leading AI labs face a growing challenge: Better models aren't necessarily attracting more usage. A 10-point intelligence advantage now earns just 20% more developer traffic, down from nearly 69% in late 2025. Full story: thein.fo/4hGJ9jY

I don’t want to manage a agents with different personas. If intelligence is truly super there should be no need for roles. Smart generalists rule the world. If you’re making agents with personas, are you actually finding value from it?

Auditing RL environments and reward models is one of the highest-leverage tasks evaluators can do in service of building aligned models (embedded or otherwise).

Agents write application code, but you still get paged at 2am when it breaks. We wanted to know whether AI could handle that part of the job too. Introducing Incident Arena: a benchmark that puts coding agents on call! Check out our paper & full dataset release below!