Member of Technical Staff @GoodfireAI
San Francisco, CA
Joined March 2022
- Tweets324
- Following652
- Followers756
- Likes1.3K
Tal Haklay retweeted
Two more weeks to apply for the @azrielifdn post-doc fellowship, an excellent program, funding, and cohort to interact with. Let me know if you're interested.
azrielifoundation.org/fellow…
Tal Haklay retweeted
Long post about my thoughts on this:
I'm so excited that this is finally out! Optimizing this until scaling it out is actually feasible and building the systems to perform the scaling has been my job for the last couple of months. This work is some of the most deserving of my time and energy that I can imagine.
There's been a lot of Discourse in the interpretability community lately about the neuronal and featurizer paradigms (not enough discussion of the causal abstraction paradigm imo..), the LRH and superposition, etc but no matter what paradigm we find ourselves in (and remember they can co-exist and describe different phenomena!) it's paramount that we accelerate the science of how models handle a wide array of tasks, how compositionality is constructed through intersection and containment of internal machinery (whether neurons, circuits, subspaces, or whatever else). Our current approaches have been too isolated to specific task and model combinations, but now that we have proven methodologies we can begin the process of "normal science" by applying them to population studies.
Once we've proven this out, it will be time to scale the effort beyond even the model-specific setting. This is where things get really interesting! Given how many different axes models vary on, we need to see what findings hold across which axes (model size, architecture, attention schemes, etc) if we are ever to develop "laws" of interpretability. Only then can we start to apply mechanistic alignment techniques broadly across the industry and have a reasonable expectation that it will work.
Personally, I'm quite optimistic based on our early results. I firmly believe that this is a solvable problem, but it won't be easy and requires an enormous amount of domain expertise (I'm so grateful to work with such talented researchers) to keep the agents from slopping it up. It will require an even more enormous about of resources, which is why it has to be an effort of the community at large. When we're ready, I hope that we can rally everyone who has the drive to solve these hard problems to use our encyclopedia and to help us fill it out.
if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
Tal Haklay retweeted
The most ambitious vision for interpretability has long been to reverse-engineer a model well enough to answer an arbitrary question about how it works.
For years, that’s seemed out of reach. Now, we think it’s possible. Neural networks are cleaner than we thought: we can now understand most neurons in models up to trillions of parameters. And agents can rapidly accelerate our progress.
But to get there, we need a concerted effort to unlock scaling laws for interpretability, where our understanding of a model grows predictably with compute.
That’s why we’re committing $10M to start a Human Genome Project for AI. Our ultimate goal is to reverse-engineer a frontier scale model and publish a living encyclopedia explaining how it works.
We don’t know exactly what we’ll find, and we can’t do it alone. We hope you’ll join us. Our chief scientist @banburismus_ lays out the vision here:
if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
Tal Haklay retweeted
if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
Tal Haklay retweeted
I'm at #EMNLP2026 in Budapest over the next few days! Presenting this work tomorrow, Oct 25, Poster Session B (16:00–17:30, Hall H).
If you're into Speech LMs, interpretability or neuro-AI, feel free to DM me to grab a coffee or chat! ✌️🤗
Tal Haklay retweeted
The average human lifespan is just four thousand weeks. Every day is precious and fleeting. That is why I am requesting that the SAC charge the authors of this paper with murder
Tal Haklay retweeted
Very happy to see this from Nick!
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
working to help get these results has been very exciting, but there are still many unanswered questions
if expanding on this line of research is interesting to you, we're currently looking for research fellows to work with me on the many followups involved! link in next tweet
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
Check on your neurons
“The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread”
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Good news from @GoodfireAI 🔥
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
yessss i can finally talk about this! neurons are much more interpretable than expected, very excited to see where this goes
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even eliezer will have to lower his pdoom. we can start making alignment into a 'normal science'
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that's the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.
Goodfire's @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there's reason for optimism today, and why there might be a bitter lesson for interpretability.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Tal Haklay retweeted
Excited to share this new result on EVEE, interp generated hypothesis actually reclassifying VUS. Want to share a bit about how this happened 🧵 (1/n)
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it successfully reclassified as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
Tal Haklay retweeted
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it successfully reclassified as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
Tal Haklay retweeted
People should pay more attention to what Goodfire is doing
David Sacks says Anthropic treating Claude like a human is the reason the AI alignment field has produced nothing useful in 10 years
"I actually think that this might be a reason why the whole field of alignment has been so spectacularly unsuccessful over the past five or 10 years. As far as I can tell it hasn't produced anything of tangible value.”
“My theory on this is that they're overcomplicating the task massively by trying to train the model to behave based on an 80-page ethical system that they've codified instead of a very simple list of rules like follow the law or just do what the user wants as long as it doesn't break the law. That's what alignment should be doing."
"But they're not doing that. They're trying to align it to an ethical system. They're treating it like it is a human, like it has its own consciousness, that it's a ‘moral patient.’”
“They are encouraging this model to have human-like self conception in terms of preferences, well being, a sense of self. Again they teach it to act like a conscientious objector who can push back or refuse instructions even from Anthropic if the model judges them to be wrong."
Tal Haklay retweeted
It was a packed day, full of interesting conversations and engagement!
Thank you to all the researchers, speakers, and panelists for making this happen.
See you next year!
Tal Haklay retweeted
Open models can come from anywhere. Securing them shouldn’t depend on where you get them.
Today we're launching Project Beacon to make safety a default for open intelligence.
We're partnering with @GoodfireAI to bring inference and in-line monitoring together for the first time, so safety signals are caught proactively.
Read more in our announcement: baseten.co/blog/safe-open-mo…
Tal Haklay retweeted
Goodfire safety infrastructure 🤝 @Baseten inference
Today we're partnering with Baseten to bring real-time safety monitoring to open models. This is an important step toward our vision of ensuring every capable model has a frontier alignment stack.
Open models can come from anywhere. Securing them shouldn’t depend on where you get them.
Today we're launching Project Beacon to make safety a default for open intelligence.
We're partnering with @GoodfireAI to bring inference and in-line monitoring together for the first time, so safety signals are caught proactively.
Read more in our announcement: baseten.co/blog/safe-open-mo…