🧪Can AI agents take a scientific paper and write a meaningful follow-up to it? We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs: @LosAlamosNatLab @SandiaLabs @BerkeleyLab @AmesNatLab @PPPLab @Livermore_Lab

Congrats to everyone with papers accepted to #NeurIPS. I am thrilled to share that we successfully withdrew one of our #ICLR submissions, because we caught a serious evaluation error before submitting the #slop. Story: We used Codex/Claude Code to help design and debug LLM evaluations on modest GPUs. When we hit OOM errors, Claude Code suggested several fixes, including reducing the max output-token budget. Tired of auditing every suggestion, we approved it. Two months later, while analyzing the results, we discovered that this “small” change substantially altered the evaluation outcomes. The reduced generation budget became a major confound and made our method look much better than it actually was. We thought we had made significant progress. BUT it was, to a large extent, an evaluation artifact. AI can accelerate research. It can also accelerate confounding. And thanks to AI who helped us confirmed the error... NOTE: Image is AI-generated for illustration only; all numbers are made up but the story is true.