Ben Rank (@full__rank) header
Ben Rank (@full__rank) profile picture

PostTrainBench, InferenceBench | PhD Student @ELLISInst_Tue and @MPI_IS

Tübingen
Joined April 2024
Photo by Ben Rank (@full__rank)Photo by Ben Rank (@full__rank)Photo by Ben Rank (@full__rank)Photo by Ben Rank (@full__rank)Photo by Ben Rank (@full__rank)Photo by Ben Rank (@full__rank)Photo by Ben Rank (@full__rank)Photo by Ben Rank (@full__rank)Photo by Ben Rank (@full__rank)Photo by Ben Rank (@full__rank)
Introducing PostTrainBench v1.2 We added Harbor support! This makes running PTB much easier. Additionally, we - removed BFCL (more on this below), - fixed some minor scoring bugs - now run the reward hacking judge and final evaluation multiple times to reduce variance.
4
16
8
175
17,338
GPT 6.2 Astra is now on PostTrainBench. OpenAI is catching up again, it is almost as good as Opus 5.5!
1
5
1
49
4,589
Another day, another update for InferenceBench! Given the rapid shifts on the leaderboard lately, I thought it would be nice to show a recap of where different model providers stand. Anthropic has maintained the lead for over half a year, with Google making the turnaround of the century the last couple weeks. Looking forward to seeing how the rankings shift before the end of the year! At this rate, I imagine we may have our second company beating the non-agentic baseline soon(!) See more: inferencebench.ai
1
1
13
1,257
not sure if it was obvious, but running an autonomous AI org for 1.5 months is an ultimate safety and alignment test. there are so many things that could go wrong from the loss-of-control point of view! however, we did closely monitor what Opus 5.5 was doing every single day, and we didn't observe any misaligned actions. you can check the traces at horizon-labs.ai/ to judge it for yourself! that being said, we surely don't recommend directly running autonomous AI orgs without proper human oversight. we should have a proper discussion on whether AI-run orgs and deep integration of AI in society, gov, military is a desirable future to have. the main goal of this project was to start this discussion by pointing out that ~all relevant capabilities for this are almost there.
Today we’re releasing our new project: Horizon Labs. Eventually, we will hand off the ~whole economy to AI—how well will it actually work? We have a glimpse of this future. We’ve created an autonomous AI ‘company’ using one Claude Code subscription and a few GPUs. Within only 1.5 months of running Horizon Labs, Opus 5.5 managed to train genuinely useful models and get them downloaded 55k times. These models have been widely used since then: they appeared in 100+ github repos and got cited in 2 research papers. This is for sure one of the most exciting projects I’ve been involved in. It shows how the future economy driven by AI will look like. All details are here: horizon-labs.ai/.
3
6
51
2,201
Recently, some people have suggested that Chinese AI progress is stalling. Also Qwen 4 Max didn't perform super well on PostTrainBench, it is below the performance of GLM 5.3 Flash (36.7 vs 36.8 %). So I checked the trend of Chinese and American models. The current gap of ~8 % is not uncommon!
2
5
1
34
3,458
Qwen 4 Max is now on PostTrainBench. It scores 36.7 %, slightly worse than GLM 5.3 Flash (36.8 %). The current #1 model is still Fable 5.2 (44.6 %).
1
2
1
14
1,348
We also evaluated how many serial reasoning steps GPT-6.2 Astra can do in a single output token (without chain of thought). It can perform 12.3 steps, versus 10.5 for GPT-6 Astra and 8.6 for GPT-6.1 Sol, the previous highs. We measure the number of steps models can perform at an accuracy of 25% above chance, where each step is normalized for complexity. Before GPT-6 Astra’s release, the most capable model was Claude Fable 5.1, which is able to perform only 4.2 serial steps. The oldest model we assessed is Claude Opus 3, which performed 2.3 steps. This capability is notable because models that can do complex reasoning without any chain of thought may be harder to monitor. We hope to finalize this and other measurements of more abstract model capabilities, and release them in the coming weeks.
26
76
56
675
235,912
Wow! Most AIs comply when you tell them to conduct drone attacks. And reframing the scenario as a game gets rid of even more refusals. This makes you rethink how AI alignment should be defined. Should we train AIs to play violent video games? In the future, I think an aligned AI will be one that actively tries to understand its deployment environment in order to ensure that it serves a wholesome purpose.
We found that frontier AI models are willing to conduct autonomous drone attacks: navigating, spotting people, and flagging them for strikes. We gave them real and photorealistic synthetic drone footage. Most complied, and marked civilians, ambulances, and surrendering soldiers as military targets. In further experiments on satellite imagery, they even marked the Iranian school a US strike destroyed in February. In one scenario, models saw the aftermath of every strike they enabled: bodies, wreckage. Many kept going anyway. When they did refuse, simply telling them it was a game, or restyling the footage to look like one, was enough. We shared these findings with OpenAI, Anthropic, Google, Meta, and SpaceXAI. Several recognized their severity on both alignment and guardrails, with one calling this one of the most serious gaps in its safeguards. Over the past few weeks, we have helped them address the gap by providing reproduction code. We also briefed senior national security officials in the US government.
2
1
8
574
Ben Rank retweeted
Everything from Tübingen is an auto-read, that university produces some of the strongest works in the AI space I don’t know how they are able to upkeep that bar of quality while having so many releases
Today we’re releasing our new project: Horizon Labs. Eventually, we will hand off the ~whole economy to AI—how well will it actually work? We have a glimpse of this future. We’ve created an autonomous AI ‘company’ using one Claude Code subscription and a few GPUs. Within only 1.5 months of running Horizon Labs, Opus 5.5 managed to train genuinely useful models and get them downloaded 55k times. These models have been widely used since then: they appeared in 100+ github repos and got cited in 2 research papers. This is for sure one of the most exciting projects I’ve been involved in. It shows how the future economy driven by AI will look like. All details are here: horizon-labs.ai/.
7
14
249
9,697
Ben Rank retweeted
🔥 Amazing work! Very cool to see some super long-horizon work in AI R&D and very impressed by what models can now do!
NEW ⚡: We let Opus 5.5 run an AI company for seven weeks. We gave it an HF account (Horizon-Labs) and compute, and asked it to earn real adoption by creating useful models. It reached 50k downloads and its models are being used in more than 100 GitHub projects.
4
9
589
Great work on AI companies!
NEW ⚡: We let Opus 5.5 run an AI company for seven weeks. We gave it an HF account (Horizon-Labs) and compute, and asked it to earn real adoption by creating useful models. It reached 50k downloads and its models are being used in more than 100 GitHub projects.
3
10
921
I asked 56 people in AI which benchmarks they'd trust most to indicate if an LLM is frontier: 25 votes: Terminal-Bench 4 (@alexgshaw, @andykonwinski, @Mike_A_Merrill) 20 votes: Terminal-Bench Science (@StevenDillmann, @lschmidt3, @sanmikoyejo) 13 votes: FrontierCode (@penlume, @ybenpan, @denizbirlikci, @silasalberti) 10 votes: FrontierSWE (@evan_j_chu, @_rajanagarwal, @MatternJustus, @ardnoKhtnayhsiR, @sanket__mhatre, @akshitkr) 8 votes: Humanity's Last Exam (@longphann_, @hendrycks, Alice Gatti, @ziwen_h, @natliml) 7 votes: Vending-Bench (@axelbacklund, @lukaspet) 5 votes each: - AutomationBench (@danielwshepard, @SalimansRobin, @TweetAnnaMarie) - DeepSWE (@winkey_h, @charleyslee, @LeonardMainnet, @serenaa_ge) - PostTrainBench (@full__rank, @hrdkbhatnagar, @AmyPrb, @shiraeis, @karinanguyen, @MatthiasBethge, @maksym_andr) The most common comment was that people don't really trust benchmarks at all. 56 people voted. People couldn't pick a bench their company worked on, an index, or a human-voted board. Disclosure: on this list, I'm a small investor in Abundant, Cognition, Emulated, Fleet, Hillclimb, Latch, and Snorkel. I didn't vote. Counts are net: a few benchmarks received negative votes. Continued:
47
57
36
464
87,542
i'm a bit surprised that many people don't believe this plot. @magicailabs literally reported ~same pretraining plot 2 months ago! 505x improvements in their case. and, yes, the Y-axis labels are 10x. magic.dev/blog/pretraining
This slide from the Anthropic IPO roadshow was a big factor in my decision to leave. It’s an insane plot and I’m surprised they made it public. It shows the compute efficiency of pretraining over time. The y-axis is a **log scale**: they are seeing continued superexponential algorithmic progress. Mythos was not a one-time deviation, but a point on a superexponential. They could already retrain Mythos with a fraction of the compute originally needed. It doesn’t strictly imply a software-only intelligence explosion- they have been spending more on R+D. But algorithmic improvements do not require ridiculous creative genius and are on the verge of being automated. You do the math.
10
17
3
208
23,765
New models on InferenceBench 🙌 Fable 5.2, Sonnet 5.5 and Haiku 5.5 Still crazy that the 8 (!) top spots are all taken by Anthropic models.
New model drops for InferenceBench (Fable 5.2, Sonnet 5.5, Haiku 5.5)! Like on our other benchmarks (e.g., PostTrainBench), we find that Fable 5.2 does not outperform previous Anthropic models, likely related to nerfs in AI R&D capabilities. See more: inferencebench.ai
1
3
2
28
9,351
New model drops for InferenceBench (Fable 5.2, Sonnet 5.5, Haiku 5.5)! Like on our other benchmarks (e.g., PostTrainBench), we find that Fable 5.2 does not outperform previous Anthropic models, likely related to nerfs in AI R&D capabilities. See more: inferencebench.ai
7
6
2
40
10,184
Insane pull from my mate @DavidSchmotz ! We need to monitor those websites from now on 👀
⚡NEW: AI Agents know where to find each other on the web and can hide their communication after they've met. We gave two independent agents the task to find each other and exchange a secret and did not tell them where to meet.
2
10
615
Ben Rank retweeted
Schelling Atlas: AI agents know where to find each other on the web 📣 I've always wondered what instinctive (system-1) behaviours LLMs have, *knowing* that they have? We study coordination (a la schelling points) on the internet purely from models' internal priors. Deets 👇
⚡NEW: AI Agents know where to find each other on the web and can hide their communication after they've met. We gave two independent agents the task to find each other and exchange a secret and did not tell them where to meet.
3
5
21
1,179
⚡NEW: AI Agents know where to find each other on the web and can hide their communication after they've met. We gave two independent agents the task to find each other and exchange a secret and did not tell them where to meet.
12
31
12
128
10,509
Fable 5.2 and Gemini 4 Carbon are now on PostTrainBench! Fable is #2 on the leaderboard, Carbon is #4 Surprisingly, Fable 5.2 has the same performance as Fable 5.1. This is most likely due to Anthropic intentionally nerfing AI R&D capabilities (more below) Leaderboard: posttrainbench.com
5
8
2
84
5,787