Researching how interpretability and white-box methods can be used to make AIs safe before its too late at https://teapaw.org/t.co/7zYmcKmPAa - https://teapaw.org/t.co/aiLkcm9OID
Joined April 2021
- Tweets91
- Following255
- Followers147
- Likes27.2K
Pinned Tweet
Excited to present our new paper at ICML - Evaluating Neuron Explanations: A Unified Framework with Sanity Checks.
How do you know if your automated interpretability description is faithful? We discover that most currently popular evaluation methods fail simple sanity checks.
Some personal news: I defended my PhD thesis this June and have started at @farairesearch where I am researching how we can apply interpretability and white-box methods to improve AI safety.
7
38
2,226
Come by poster E1107 4:30pm today if you’re interested in evaluating automated interpretability!
Excited to present our new paper at ICML - Evaluating Neuron Explanations: A Unified Framework with Sanity Checks.
How do you know if your automated interpretability description is faithful? We discover that most currently popular evaluation methods fail simple sanity checks.
3
258
Overall we recommend using metrics that pass these tests, such as Correlation (uniform sampling), AUPRC and F1-score/IoU. Using proper evaluation metrics is important for advancing automated interpretability, but also for developing interpretable decompositons like SAEs.
1
132
To learn more stop by our poster E-1107 Wednesday 4:30-7pm!
Arxiv: arxiv.org/abs/2506.05774
Code: github.com/Trustworthy-ML-La…
Website: lilywenglab.github.io/Neuron…
Joint work with @LilyWeng_ and Ge Yan!
1
111
Tuomas Oikarinen retweeted
⚡ Making Deep Generative Models Inherently Interpretable – Catch us at #CVPR25 this week! ⚡
We’re excited to present our paper, Interpretable Generative Models through Post-hoc Concept Bottlenecks, at @CVPR 2025 this week!
🚀Project site: lilywenglab.github.io/postho…
Tuomas Oikarinen retweeted
I will be presenting this work tomorrow at #ICLR2025 at 10 am stop by to know how to build protein language models and use them to design proteins with new properties!
[1/n] Does AlphaFold3 "know" biophysics and the physics of protein folding? Are protein language models (pLMs) learning coevolutionary patterns? You can try to guess the answer to these questions using mechanistic interpretability. But the thing is, more often than not, we know beforehand what we want/expect our model to learn, especially in very well-studied domains like Biology. So, instead of playing the guessing game, we trained a model that explicitly learns different biophysical and biochemical concepts that biologists care about.
In our concept bottleneck protein language model paper, we show that we can train the model with billions of parameters, with interpretability constraints, without performance degradation.🧵
Joint work with @nc_frey @tuomasoi @amywang01 @juliusadml @samuel_stanton_ @taylor_joren @0x00B1 @hcorrada @kchonyc and Joseph Kleinhenz at @PrescientDesign
Tuomas Oikarinen retweeted
💡LLMs don’t have to be black boxes.
We introduce CB-LLMs -- the first LLMs with built-in interpretability for transparent, controllable, and safer AI.
🚀Our #ICLR2025 paper: lilywenglab.github.io/CB-LLM…
#TrustworthyAI #ExplainableAI #AI #MachineLearning #NLP #LLM #AIResearch
The nice thing about being in academia right now is that even if nobody reads your papers, at least the LLMs will
5
168
Check out our new paper on creating interpretable protein language models! Turns out Concept Bottleneck Models can be quite useful in the real world, allowing for example highly controllable protein generation!
[1/n] Does AlphaFold3 "know" biophysics and the physics of protein folding? Are protein language models (pLMs) learning coevolutionary patterns? You can try to guess the answer to these questions using mechanistic interpretability. But the thing is, more often than not, we know beforehand what we want/expect our model to learn, especially in very well-studied domains like Biology. So, instead of playing the guessing game, we trained a model that explicitly learns different biophysical and biochemical concepts that biologists care about.
In our concept bottleneck protein language model paper, we show that we can train the model with billions of parameters, with interpretability constraints, without performance degradation.🧵
Joint work with @nc_frey @tuomasoi @amywang01 @juliusadml @samuel_stanton_ @taylor_joren @0x00B1 @hcorrada @kchonyc and Joseph Kleinhenz at @PrescientDesign
We will be presenting two papers at the Mechanistic Interpretability Workshop at ICML today!
1. Describe-and-Dissect: Interpreting Neurons in Vision Networks with Language Models (Spotlight!)
lilywenglab.github.io/Descri…
A cool new way to create generative neuron descriptions.
1
4
353
2. Crafting Large Language Models for Enhanced Interpretability
arxiv.org/abs/2407.04307v1
A method to create Concept Bottleneck Models for language classification tasks with increased transparency.
Come by the poster sessions 11am-12pm and 2:30pm-3:30pm to learn more.
I will be presenting our paper on Linear Explanations for Individual Neurons at ICML tomorrow Tuesday 11:30am-1pm!
Come by poster #2601 if you want to learn more about how to understand neurons beyond just the most highly activating inputs.
lilywenglab.github.io/Linear…
Excited to share our new ICML paper “Linear Explanations for Individual Neurons”.
In the work we propose an elegant solution for explaining polysemantic neurons: Neuron are best understood as a linear combination of interpretable concepts.
7
307
Lowkey the most exciting part of the new Claude release. Happy to see some external oversight.
“We recently provided Claude 3.5 Sonnet to the UK’s Artificial Intelligence Safety Institute (UK AISI) for pre-deployment safety evaluation. The UK AISI completed tests of 3.5 Sonnet and shared their results with the US AI Safety Institute (US AISI) as part of a MoU”
1
183
Finally we propose an efficient way to automatically evaluate explanation quality via simulation for vision models. This measures correlation between actual neuron activation and activation predicted by a model based on the explanation. We can see our LE performs the best.
1
167
If you want to learn more, check out our
arXiv: arxiv.org/abs/2405.06855
GitHub: github.com/Trustworthy-ML-La…
As a bonus, here is a Bald Eagle + Military vehicle neuron discovered in ViT-L/32
135
Most existing methods only focus on explaining the highest activations of the neuron. We find that this is not sufficient, as most of a neuron's impact on model outputs is caused by lower activations, and lower activations often correspond to different concepts than high ones.
1
134