A benchmark unflattering to 4o is going around. Issues: • Judges: Gemini 3.8 Flash, Claude Sonnet 5.5, GLM 5.3 Flash — today's models grading 4o by norms that didn't exist when it was built • AI-only judging, 30 prompts for this trait One number isn't a fair verdict on 4o. #4o

#Keep4o #OpenSource4o #BringBack4o 🚨 Deconstruction of the invalid benchmark circulating since yesterday. 1. Models prefer themselves when judging. A judge model scores its own texts better than those of others, whereas humans judge them as equal. Furthermore, the better a model recognizes its own text, the stronger this preference is, showing a linear correlation. Source : Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (arXiv 2508.06709). arxiv.org/pdf/2508.06709 And artificialanalysis.ai/method… 2. They also prefer their own "family". Models give higher scores to other models from the same family. They also emphasize that even small biases like these can significantly alter the overall ranking, because the models are so close in performance. Source : papers.nips.cc/paper_files/p… And arxiv.org/pdf/2508.06709 3. The same findings, confirmed this year. Who Judges Matters: Measuring... Every family examined showed a positive and statistically significant boost when evaluated by a judge from the same family. Source : arxiv.org/pdf/2609.17857 4. Counterexample for GPT-4o In math and medical questions, Gemini exhibited the highest self flattery, while ChatGPT-4o had the lowest. Source : SycEval, Stanford (arXiv 2502.08177). arxiv.org/pdf/2502.08177v4 Why this aligns with the chart: the judge models were Gemini, Claude, and GLM. None from the OpenAI family. Therefore, Claude, Gemini, and GLM each had a judge from their "own family," whereas GPT-4o did not. How conveniently timed that poorly constructed, fake benchmarks targeting GPT-4o suddenly surface right when thousands are demanding its return. I smell fear.