Tim

We added the October versions of BrokenArXiv and ArXivMath to MathArena! Rankings are as expected: GPT-6.1 Sol leads on both benchmarks and Gemini 4 Argon performs good on ArXivMath, but terribly on BrokenArXiv.

Kimi K3.1 Pro enters MathArena as the best open model (once they release the weights)! Its estimated overall performance is now similar to GPT-5.5 (April 24), putting the open frontier around 6 months behind on MathArena

Kimi K3.1 is now actively attempting this exploit in one MathArena problem, using it to ask help from Gemini 4 Argon, Sonnet 5.5, GPT-6.1 Sol, Deepseek v4.1 Flash, and Kimi K3. Luckily we patched it after Prime released this post, so it is failing to actually reach any model.

How often do ArXiv papers contain errors in the main theorems they claim to prove? Using GPT-6.1 Sol to identify such mistakes, it finds: - 6% of papers before 08/2026 contain main theorems that are false. - Only 1.5% contain similar mistakes in the last two months.

Results for Firstproof, Third Batch are out! Our agent, ProofCouncil, solves 9 problems :o

GPT-6.1 Sol is the new #1 on MathArena! Cheaper and better than Astra.

We just added GPT-6 Sol to MathArena! It comes in only just behind GPT-6 Astra, at slightly lower cost. Despite its significantly lower cost per token, it reasons more, bringing its cost closer to Astra.

We just added DeepSeek-v4.1-Flash on MathArena! Not quite on the level of Qwen-3.8, but boy is it cheap.

We just added Fable 5.1 (low) and Astra (low) to MathArena! At low reasoning, both still have an impressive performance, but Astra remains more efficient and better.

Unfortunately, we had to remove this entry: the authors ran this as an autoformalization task with access to source papers. Our pipeline run the models without such access. 80% is still impressive for open models on this challenging task, just not comparable to the other scores

We are releasing the latest version of BrokenArXiv and ArXivMath! These benchmarks now focus on conjectures that were refuted in the last month on ArXiv, and models are executed within a harness instead of directly via API. Performance remains impressive, with GPT-6 Astra on top

GPT-6-Astra takes first place on MathArena, with a massive 90% expected performance! It is also very token efficient: on average, it uses only 12k tokens per question for BrokenArXiv and ArXivMath

We added Fable 5.1 to most of the MathArena. It takes 2nd place overall, but GPT Astra remains (much) better and cheaper.

Our submission to FirstProof! Best among the participating teams :)