AI scores 90% on one math test and 13% on another—the benchmarks are the problem When we test an AI model on a benchmark, we assume the score tells us something meaningful about its capabilities. But what if the benchmark is measuring the wrong things—or several things at once without knowing it? And what if a 90% score on one math test tells us almost nothing about performance on a harder math test from the same domain? That's exactly what happens. DeepSeek-R1-Distilled-Qwen-14B scores 90.5% on GSM8K but only 13.3% on OlymMATH Hard—when both supposedly test mathematical reasoning. The answer reveals a deep flaw in how we evaluate AI: most benchmarks don't measure what they claim to measure, and their aggregate scores have little predictive power for unseen tasks. Lexin Zhou and coauthors attack this head-on. They introduce 18 general rubrics—the DeLeAn rubric set—spanning cognitive capabilities like logical reasoning, metacognition, verbal comprehension, and knowledge in formal, natural, and applied sciences. Each rubric assigns a demand level from 0 to 5+ to any task instance. These levels are absolute, not relative to a population of models or benchmarks. Running GPT-4o as an automated annotator across 16,108 instances from 20 benchmarks, they build the ADeLe battery: a unified representation of task demands across 18 dimensions. They then run 15 LLMs—from Babbage-002 to OpenAI o1 and DeepSeek-R1-Distilled—to derive ability profiles: 18-dimensional vectors describing what each model can reliably solve at each demand level. The results are revealing. Knowledge dimensions scale primarily with model size. But quantitative reasoning, logical reasoning, and—strikingly—mind modeling and social cognition are boosted specifically in chain-of-thought reasoning models, even in smaller distilled variants. Most benchmarks fail basic construct validity: lacking either sensitivity or specificity, inadvertently measuring unrelated factors like question rarity or answer format. The predictive power is the key result. A simple random forest trained on demand annotations achieves AUROC ~0.84 at a fraction of the compute of fine-tuned baselines—and degrades far less in out-of-distribution settings, precisely the regime that matters for real deployment. This changes how model selection should work. Rather than relying on aggregate benchmark scores that may not transfer to your specific task, demand-based profiling lets you anticipate whether a model will succeed on literature annotation, reaction prediction, or experimental design before committing to a full pipeline integration—enabling smarter model routing and more reliable AI deployment in applied research. Paper: Zhou et al., Nature (2026): nature.com/articles/s41586-0…