Pinned Tweet
Check out this work co-led by @sophielwang. We find that base models can match much of RL’s reasoning gains with the right first tokens, which we call 𝘵𝘰𝘬𝘦𝘯 𝘤𝘶𝘦𝘴. The effective cue differs across models and can be as simple as “.\n\nOkay,” or “To determine,” with no explicit instruction to reason. We dissect this phenomenon through a set of interventions on RL and the base-model training data itself. We also extend our analysis to a safety case study of refusal and compliance.
Token cues can arise from simple associations in reasoning-related training data, and we can even turn the word “chicken” into a reasoning cue through counterfactual mid-training runs.
Super excited to find a rich underlying phenomenology! Check out Sophie’s thread for the main findings. I’ll share a few other observations below that I found especially cool. 👇
Can “chicken” make a base model reason? 🐔
Yes!
Why? With the right first tokens, base models can match RL in reasoning. We trace this to learned training data associations and use a simple data edit to make “chicken” elicit reasoning too.
blog: sophielwang.com/cues
Amil Dravid retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
Amil Dravid retweeted
Checkout this work that I'm really excited about, a massive effort by @liusida2007 & @feijianghan!
Turns out the ICA baseline in the original SAE paper had a bug. If you fix it, ICA becomes extremely competitive at finding interesting and interpretable LLM activation directions!
Amil Dravid retweeted
1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?
We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.
In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.
📝 xingjianbai.com/reactive-opt…
💻 Try it on your task: github.com/xingjian-bai/opal
🧵
Amil Dravid retweeted
I know I'm obsessive, but we have all been resizing images wrong. The way we do it is neither optically faithful nor internally consistent. Even two ×½ resizes generally don’t equal one ×¼ for crying out loud!
Building on ideas from computer vision, signal processing and optics, I derive a unified theory of image scaling connecting the clues that have been bugging me for years. >>
If you're interested in advancing our scientific understanding of deep learning, apply to @FlorentinGuth!
I'm delighted to announce that I'll join @MITEECS as an Assistant Professor next Fall! I will be recruiting graduate students this cycle, please apply if you're interested in the science of deep learning! Despite the ambient doomerism, now is a great time to do research.
Amil Dravid retweeted
Fantastic post citing some great scientists and links to their work on neuron interpretation!
Interesting food for thought, and I hope people will take interpreting MLP neurons more seriously. But “recursive generalization” vs. superposition seems to come down to whether the concepts sharing a neuron are related, and if the method is “look harder until the activations share a story,” you’ll always find one. I agree with @neelnanda that I would classify a fair amount of the examples in the post as polysemantic.
I do, however, think the essay’s proposal to understand how neurons branch out throughout training could be fruitful. But as an interpretability community, we really need to go beyond vibes and be rigorous about developing evaluations as well as toy models and theory that are predictive of real phenomena.
Due to the revived interest in interpreting neurons, I wanted to highlight some papers relevant to this discussion following @aryaman2020's post:
@davidbau et al pnas.org/doi/abs/10.1073/pna… object detector neurons in CNNs
@AlecRad et al. arxiv.org/abs/1704.01444 sentiment neuron
@YGandelsman et al arxiv.org/pdf/2406.04341 text-based interpretation of neurons in CLIP
@alesstolfo et al arxiv.org/pdf/2406.16254 neurons that modulate confidence in language models
@TamarRottShaham @cogconfluence et al. arxiv.org/pdf/2404.14394 automated interpretability of vision neurons through agents
me @YGandelsman arxiv.org/abs/2306.09346 and arxiv.org/abs/2606.03990 universal "Rosetta neurons" and how they scale with model size
@YizhouLiu0 et al. arxiv.org/pdf/2505.10465 how superposition can affect neural scaling laws
@aryaman2020 @ZhengxuanZenWu et al arxiv.org/pdf/2601.22594 circuit tracing in the neuron basis
@wesg52 et al. arxiv.org/pdf/2305.01610 finding interpretable neurons with sparse probes
@gabeeegoooh et al distill.pub/2021/multimodal-… CLIP neurons responding to multimodal concepts.
among many others
Amil Dravid retweeted
Really nice work! This supports some of my ad-hoc observations and intuitions from working on evolutionary systems and self-verification over the past year.
- While experimenting with verifier-free evolution methods combining parallel and sequential reasoning (RSA/PDR: arxiv.org/abs/2509.26626, arxiv.org/abs/2510.01123), I found that truncating the reasoning traces and keeping only the final answers (or a summary + final answers) before aggregation (roughly analogous to the communication step in multi-agent systems) often worked better. IIRC, @sudomonish also noticed something similar with SqueezeEvolve (arxiv.org/abs/2604.07725). We also observed that removing the reasoning traces made models reason for longer during aggregation, whereas retaining them made models more conservative and more likely to terminate early. This seems closely related to the papers finding that forgetting increased median reasoning tokens from 2.9k to 9.4k per proposal and leads to larger code edits as well.
- While working on self-verification (our V1 paper: arxiv.org/abs/2603.04304v1), I also noticed that self-verification performance improved when we removed the original reasoning trace and gave the model only the final answer to verify. My intuition was that models become more biased toward their own answers when they see the reasoning that led to those answers. This is consistent with @Jacklu_me's paper on verification (arxiv.org/abs/2512.02304), which shows that models are more likely to accept solutions resembling their own, while cross-family verification tends to work better. Truncating the original reasoning trace might make the answer feel more "alien" to the model during self-verification, potentially reducing this bias. The model might then spend more tokens independently verifying the answer rather than relying on the earlier reasoning. Interestingly, autoresearch agents also might be implicitly or explicitly self-verifying their own ideas and proposed changes, even when external evaluators score the resulting candidates. So I wonder if better self-verification after forgetting might also be contributing to the improvements observed in the autoresearch loops of the paper.
Amil Dravid retweeted
corroborates with our findings in arxiv.org/abs/2606.03990
Amil Dravid retweeted
Whatever you make of already-clean.goodfire.com/, it has brought compositionality and generalisation back into the spotlight. I’d like to see more rigorous work testing these ideas.
The Random Hierarchy Model is one example. By controlling the data's compositional structure, we recover several key features of deep learning: advantage of depth, scaling laws, linear representations and circuits.
Well-motivated toy models are invaluable: they help us explain how these phenomena fit together and make falsifiable predictions about real data.
Fun fact: @nickcammarata and colleagues’ earlier “Zoom In” work inspired the RHM, back when we were still thinking about cats and dogs.
Amil Dravid retweeted
1/ RL for improving task success has made LLMs remarkably capable. But what happens when the model fails? Turns out, training for graceful failure can be surprisingly tricky!
Very excited to share our new paper: Training Language Models to Fail Gracefully
🧵
Amil Dravid retweeted
REINFORCE is susceptible to getting stuck in local optima of the reward landscape. GracefulRL investigates this and provides a principled fix! Check out @damanarora’s thread for more details
Interesting food for thought, and I hope people will take interpreting MLP neurons more seriously. But “recursive generalization” vs. superposition seems to come down to whether the concepts sharing a neuron are related, and if the method is “look harder until the activations share a story,” you’ll always find one. I agree with @neelnanda that I would classify a fair amount of the examples in the post as polysemantic.
I do, however, think the essay’s proposal to understand how neurons branch out throughout training could be fruitful. But as an interpretability community, we really need to go beyond vibes and be rigorous about developing evaluations as well as toy models and theory that are predictive of real phenomena.
Due to the revived interest in interpreting neurons, I wanted to highlight some papers relevant to this discussion following @aryaman2020's post:
@davidbau et al pnas.org/doi/abs/10.1073/pna… object detector neurons in CNNs
@AlecRad et al. arxiv.org/abs/1704.01444 sentiment neuron
@YGandelsman et al arxiv.org/pdf/2406.04341 text-based interpretation of neurons in CLIP
@alesstolfo et al arxiv.org/pdf/2406.16254 neurons that modulate confidence in language models
@TamarRottShaham @cogconfluence et al. arxiv.org/pdf/2404.14394 automated interpretability of vision neurons through agents
me @YGandelsman arxiv.org/abs/2306.09346 and arxiv.org/abs/2606.03990 universal "Rosetta neurons" and how they scale with model size
@YizhouLiu0 et al. arxiv.org/pdf/2505.10465 how superposition can affect neural scaling laws
@aryaman2020 @ZhengxuanZenWu et al arxiv.org/pdf/2601.22594 circuit tracing in the neuron basis
@wesg52 et al. arxiv.org/pdf/2305.01610 finding interpretable neurons with sparse probes
@gabeeegoooh et al distill.pub/2021/multimodal-… CLIP neurons responding to multimodal concepts.
among many others
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Amil Dravid retweeted
This is interesting work, but I don't think it's a significant update to the big picture of interpretability.
I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point.
I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes!
Similarly, my impression is that SAEs generally don't find things that are a
single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework.
And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Amil Dravid retweeted
MLP neurons are great, but knowing this alone does not solve interpretability.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Amil Dravid retweeted
Superposition is not a "bug" but a "feature" driven by the fact that the world is "sparse", which may explain strong generalizability or few-shot learning: models do not ignore rare features in the training set that are frequent in some test sets. Yet, interpretability is harder
I guess there's an "interpretability phase transition".
Abstraction is simply one phase where interpretability happens to be easy.
@YizhouLiu0's work reveals a width-driven phase transition of superposition which might be super relevant here: arxiv.org/abs/2609.36455
Amil Dravid retweeted
Additionally, there is a large body of academic work (including mine) that has gradually and cautiously added to evidence that MLP neurons are an interpretable basis. It's been a bit of a heterodox line of work historically so it'd be good to see them get due credit. They surely support the conceptual claims of the blogpost:
- 2020: @megamor2 et al. arxiv.org/abs/2012.14913 MLPs as KV memories
- 2021: Damai Dai et al. arxiv.org/abs/2104.08696 "knowledge neurons"
- 2023: @_AmilDravid et al. arxiv.org/abs/2306.09346 "rosetta neurons"
- 2024: @YGandelsman et al. arxiv.org/abs/2406.04341 gradient effects of vision model neurons
- 2024: @damichoi95 @vvhuang_ et al. transluce.org/neuron-descrip… autointerp on LM neurons
- 2025: me @ZhengxuanZenWu @TransluceAI transluce.org/neuron-circuit… arxiv.org/abs/2601.22594 neuron basis is sparse in LMs
- 2026: @yaboilyrical et al. arxiv.org/abs/2605.12290 steering refusal w/ neurons
(not comprehensive by any means!)
Amil Dravid retweeted
New #NVIDIA paper: what if robots never had to wait for their own thoughts?
TacitVLA learns to reason in chain-of-thought or subtasks, but its actions never read them.
→ Keeps the gains of reasoning at up to 3.4× lower latency on 4 benchmarks, and tops RoboMME and RMBench 🧵👇
Amil Dravid retweeted
🧵[1/N] Are large language models more creative than humans at combining ideas? 🤔
Combinatorial creativity involves combining familiar ideas in unfamiliar ways, and is central to scientific discovery and technological innovation.
In fact, breakthrough scientific work is often associated with more distant combinations of methods, techniques, and concepts.
Here, we introduce Kombine, a new benchmark of combinatorial creativity for LLMs.
TLDR:
- LLMs are more combinatorially creative than the average human.
- However, conceptual blending is the hardest task for LLMs, and even frontier models struggle.
- LLMs also tend to homogenize, inventing concepts with the same properties across separate models.
- Moreover, humans and LLMs combine in distinct regions of semantic space.
- As a result, we anticipate greater conceptual diversity can be unlocked through human-AI collaboration. 🗣️
Thread below🧵⬇️
