Cam Kelley

Will scaling reasoning models like o1, o3 and R1 unlock superhuman reasoning? I asked Gwern + former OpenAI/DeepMind researchers. Warning: long post. As we scale up training and inference compute for reasoning models, will they show: A) Strong general reasoning skills that work across most logically-bound tasks B) Some generalization to other logic tasks, but perhaps requiring domain-specific retraining or C) Minimal generalization? Finbarr Timbers (Artfintel, former DeepMind) "We'll achieve superhuman performance on specific tasks with verifiable rewards. I see no evidence for general broad transfer, but it seems extremely plausible." Gwern Branwen ""Everyone neglects to ask, what are we scaling?" Depends on what data they scale up on. The more you scale up on a few domains like coding, the less I expect transfer, as they become ultra-specialized. Far transfer is rare, and after a certain point, meta-learning collapses into learning for additional optimality. AlphaZero doesn't develop general gameplaying ability that transfers strongly across all domains either." Jacob Buckman (Co-founder at Manifest AI, former Google Brain) "Generalization can't really be predicted like that except empirically. All I know is as you add more compute and data you go from minimal transfer to some transfer to broad transfer. I have no clue where on that spectrum we stand when we run out of compute or data" Karthik Narasimhan (Sierra AI, former OpenAI, co-author of GPT paper) "I expect some generalization with domain specific retraining" Near (Independent) "I think the "spikiness" of intelligence will continue to be notable (models which are extremely good at some things yet quite 'dumb' at others), but that it is easy to improve generalization in the areas we care about, since it just requires some data/RL fun." Nathan Lambert (Post-Training Lead at Allen AI) "With chain of thought generation as a general approach for autoregressive language models to break down and process information, the new models trained to reason heavily about every subject will come to have better average performance than standard autoregression. In domains with explicit verifiers, this performance will be superhuman, in domains without, reasoning will still enable better performance, but maybe not more economical performance." Pengfei Liu (Shanghai Jiao Tong University) "Increased compute and inference time will drive reasoning capabilities to expert-level performance where rich feedback loops exist. However, the development of general reasoning will be gated by two factors: the availability of problems requiring genuine deep thinking, and access to high-quality expert cognitive process data or well-defined reward signals." Ross Taylor (Previously led Meta AI reasoning team) "I think general reasoning will come fairly quickly. Right now it's easier to scale in domains where problems are easy to verify with an external signal. The generalisation will come if models themselves become good verifiers across domains." Shannon Sands (Nous Research) "There's at least some generalisation to other tasks like logic puzzles - but it might require more domain specific training to improve on many more out of domain tasks." Steve Newman (Co-founder of AI Soup, Co-founder of Writely aka Google Docs) "This is a trillion dollar question. If I had to guess: we'll see some transfer of reasoning skills across domains, but (on anything resembling current architectures) some specialized training will be needed in each domain. We'll learn a lot one way or another this year." Tamay Besiroglu (Co-founder of Epoch AI) "I think minimal transfer is wrong because reasoning is a very general skill that you can apply to perform a wide range of actions. Planning, for instance, is something that requires good reasoning, and is useful to achieve high levels of performance across a wide range of tasks. Differentiating between broad transfer and some transfer is hard without being more precise or quantitative" Teortaxes (Independent) "I think there will be a period of strong 'natively verifiable reasoning overhang' which translates to more general verifiers using models' strong coding ability and general knowledge+tools, then they grok more general regularities of sound reasoning, and the next generation can natively generate good reasoning data for all domains." Xeophon (Independent) "We will see some generalizations into other domains where the model was not explicitly trained on. For example, R1 writes better and more creative stories than V3, the model it is based on. To push this further, models need to be trained on more data in other domains." Chris Barber (Independent, Creator of this roundup) Synthesis: "The expert takes point to generalization for all logically bound domains where we can construct verifiers for now, and trending in the direction of broad transfer in the future." Implications and follow-up questions Gwern "The important question is to what extent the verifiers or judge models can be cheaply set up for each new domain, I'd think. If you can quickly 'o3-ize' every important domain, particularly with general-purpose coders & mathematician expert models, then the scaling can continue cheaply." Gwern's recent comment on LW and McLaughlin's essays are also great reads. Ross Taylor "Verification will become a test-time compute problem -> the longer you spend checking a solution, the more accurate the verification signal should be. Then the question will be: how do you verify the verifiers? I suspect we'll end up in a world where multiple agents are checking each others' work (a bit like how the research community works). Not clear if we're at the level of model capability for this to work yet, but I wouldn't bet against it!" A Separate Question - Is There a Ceiling? Finbarr Timbers "I see no reason for reasoning models to plateau around world-expert level tbh. The existing paradigm for reasoning models looks like RLVR (reinforcement learning with verifiable rewards), where the model learns to solve tasks with deterministic/verifiable rewards. No reason that has any limit around human levels. Look at deepseek r1, for instance." Jacob Buckman "In general search can surpass humans. This was clear even pre-deep-learning, e.g. with "classical" chess AI. But most real-world domains don't have the nice properties (cheap to simulate, clear success condition) that make them amenable to search. In a setting without this niceness, reasoning/search amounts to learning a human expert's "search function", and then unrolling it. It's possible to surpass the original expert using this technique, by unrolling for more steps than they do. The bottleneck in this case becomes rate-of-search, that is, you won't be able to improve performance faster per step of search than the expert." Tamay suggested a good follow up would be to "elicit ideas for experiments that people would expect to turn out one way conditional on “Weak transfer” and another if “Strong transfer” is correct" – let me know if you have ideas. Thank you to Amir Haghighat, Arun Rao, Ash Bhat, Avery Lamp, Charlie Songhurst, Connor Mann, Daniel Kang, Dhruv Singh, Eric Jang, Ethan Beal-Brown, Finbarr Timbers, Flo Crivello, Griffin Choe, Gwern, Herrick Fang, Jacob Buckman, James Betker, Jay Hack, Josh Singer, Julian Michael, Katja Grace, Karthik Narasimhan, Logan Graham, Matt Figdore, Mike Choi, Nathan Lambert, Nicholas Carlini, Nitish Kulkarni, Pengfei Liu, Rick Barber, Robert Nishihara, Robert Wachen, Rohit Krishnan, Ron Bhattacharyay, Ross Taylor, Shannon Sands, Spencer Greenberg, Steve Newman, Tamay Besiroglu, Teknium, Teortaxes, Tim Shi, Tim Wee, Tom, Tyler Cowen, and Xeophon. Upcoming posts: 1) Computer-use benchmark timeline predictions 2) Clips from researcher interviews (e.g. Finbarr Timbers and Jacob Buckman from the list above) 3) More takes on scaling reasoning models Questions: 1. What's your take? 2. Who else should I ask? 3. What follow up questions should I ask? 4. What's the even more important question that I should be asking instead?