AI engineering at Magentic | ex-PhD @AIMS_Oxford @OxfordTVG @UniofOxford, ex-@_FiveAI (views are my own)
London
Joined September 2012
- Tweets140
- Following347
- Followers281
- Likes855
Pinned Tweet
🧵 Part 2 of testing the experiments behind “Neural Networks Are Already Clean”. Its language-model example: four neurons in Qwen3-32B called a preeclampsia “diagnosis hub”. I pre-registered tests on 160 prompts. They’re a real preeclampsia signal, but not a diagnosis hub 👇
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
2
107
8/ 📝 Post, with interactive figures: fgirbal.github.io/blog/preec…
1
6
See also the original thread on InceptionV1's "V neuron": teapaw.org/fgirbal/status/2111775…
🧵 Interpretability stories are easy to tell and hard to test. This essay argues InceptionV1’s “V neuron” detects pointed shapes. We ran pre-registered, first-principles tests: mostly not. Keep a photo’s outline, wipe the inside, and the response collapses.
Why it matters 👇
6
🧵 Interpretability stories are easy to tell and hard to test. This essay argues InceptionV1’s “V neuron” detects pointed shapes. We ran pre-registered, first-principles tests: mostly not. Keep a photo’s outline, wipe the inside, and the response collapses.
Why it matters 👇
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
9/ 🌍 Is neuron-level interpretability solved? No. And I think it’s only a small part of trusting a model anyway. What’s missing is interpretability with skin in the game: like in a prediction market, a claim is worth what it delivers. Guardrails that hold, unlearning that sticks
1
1
52
10/ 📝 Full post with interactive figures: fgirbal.github.io/blog/v-neu…
1
3
51
3/ 🧭 The essay’s claim: pointed things drive it, round things push it negative. So a pointedness detector should (a) respond to a V on its own, (b) rely on the outline, not what’s inside, (c) prefer a V when the content is fixed. I pre-registered these, with thresholds.
1
11
