The gap between frontier models' ability to derive clearly superhuman results and their absolute lack of ability to write about them in a human legible way is IMO the best current example of the "jagged frontier"

As an author of one of the 15 audited benchmarks, some personal thoughts on this: 1. There’s such a tight loop today between measurement and optimization, that high profile benchmarks materially impact what counts as progress in our field. This really raises the stakes on benchmarking work, and makes the difficult and not very glamorous work of re-examining existing benchmarks with critical (and impartial) eyes extremely, extremely valuable. I’m really glad to see @EpochAIResearch starting this initiative, congrats. 2. On the other hand, as someone working on frontier evals for the past few years, I want to emphasize how virtually impossible it is to put out a flawless benchmark. And it gets meaningfully harder the wider the scope of the benchmark is and the more agentic the task is. 3. I don't think it's helpful to view benchmarks as binary good vs bad, “flawed” vs “verified”. Auditing them, finding problems and improving on those problems is better viewed as a continuous process. In some sense it inevitably is, because some grading problems only become apparent when models become capable enough to utilize them, and are really hard to forsee in advance. 4. The wider the participation in this process will be, and the more parties that have a stake in these benchmarks, the better the outcome will be for the entire community. 5. This is essentially what we tried to do with SimpleQA Verified: we used OpenAI’s SimpleQA internally for some of our factuality work, realized it has a lot of issues, improved on them as much as we could at the time, and put the revised benchmark back out in the hands of the community. Based on my personal experience working with some of the other benchmarks audited by Epoch AI that were found as problematic, this can work just as well on them too.