When will robotics be solved?
We put frontier models in real-time MuJoCo sims of a robot arm and a humanoid, where they only receive camera feeds and sensor readings, and respond with motor commands. We tested them on 24 tasks built around spatial reasoning and reaction time.
Oliver retweeted
RSI in ~270 days so best of luck
A new White House task force will have 120 days to report on the risks of artificial intelligence and come up with decisions on what responsibility the federal government has, @learyreports with the exclusive wsj.com/tech/ai/new-ai-task-… via @WSJ
Massive jump from Google on our RSI-Index. Now #4 and outperforming GPT-6-Astra. Were the internal RSI rumors true 👀? vals.ai/benchmarks/rsi_index
Tons of infra work went into this! Congrats to @hungtran and team for getting this out
Oliver retweeted
Fable solved the Cyphral Distich (a 370 year old cypher). Super cool way to use Claude
vals.ai/blogs/fable-solves-c…
Oliver retweeted
“What would we even do during an AI slowdown?”
Containment. It will take at least a year of dedicated work to harden security to ensure AIs can't self-exfiltrate, and that adversarial nations can't steal the weights of cyber-offensive AIs [1].
Propensities. Capabilities (what an AI can do) are different from propensities (what it tends to do). We can work on improving AI propensities to ensure they have a negligible rate of lying, cheating, and wanton harm.
Adversarial robustness. We can also harden AIs against jailbreaks, prompt injection, and backdoors. Obtaining high levels of robustness requires careful, assiduous work, as with autonomous vehicles.
Institutional adaptation. We have to greatly increase state capacity to understand and manage AI. Communities also need time to figure out how to handle AI (like AI in education). Civil society also needs to be diversified: nearly all funding for AI safety organizations is directed by the EA/utilitarian network [2]; risk management needs more independent funders, values, and centers of power.
Moonshots. We can explore different paradigms for safety: mathematical foundations [3], neuroscience-based interpretability [4], safe-by-design architectures [5], and beyond.
AI for good. We can collect targeted post-training data to make AI exceptional at radiology, weather forecasting, agriculture, and so on. Fortunately, we can detect if data or avenues of research actually target beneficial use cases or just secretly push general capabilities [6].
A slowdown means we don't have to bet the species to capture the benefits of AI.
Oliver retweeted
100%, this is going to be one of the most important roles in the world - the people will need to be of incredible integity and technical expertise, and be drawn from a wide enough set of backgrounds that all of society feels confidence in their judgements.
Today, we're launching Vals Public Sector: independent AI evaluation for government.
Recent events have made one thing clear: the government has a stake in understanding how frontier AI models perform on day-to-day work, and the risks they carry.
This work is central to who we are. We've spent years building industry-leading benchmarks alongside the top AI labs and enterprise domain experts. Now, we’re bringing that same rigor to support government where AI matters most, from public benefits to national security.
We are excited for the work ahead!
We’re releasing our Code Migration benchmark — and we managed to get Fable tested in time
Code migration carries real economic weight. COBOL powers banks, payrolls, government services, and underpins nearly 95% of US ATM transactions. The danger with any migration is that a model ships code that looks right but quietly drops essential behaviors
We evaluated models in three ways: modern to modern migrations, legacy to modern migrations and on their overall code quality. Each model rebuilds the program in an offline sandbox, then is scored on a hidden behavior test with anti-cheat checks that catch anything wrapping the original, copying reference files, or staying in the source language
Fable 5 leads overall at 55%, but costs $115.43 per test, while Opus 4.8 (47%), and GPT 5.5 (45%) cost $30.51 and $6.44, respectively, making GPT 5.5 the most cost efficient model. Kimi K2.6 is the #1 open-weight model (28%) priced at $5.12, ranking above some frontier models




















