Interpretability @GoodfireAI | Physics | Evolution | Ecology
Joined September 2015
- Tweets131
- Following648
- Followers325
- Likes743
Michael Pearce retweeted
Excited to share this new result on EVEE, interp generated hypothesis actually reclassifying VUS. Want to share a bit about how this happened 🧵 (1/n)
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it successfully reclassified as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
Michael Pearce retweeted
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it successfully reclassified as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
Michael Pearce retweeted
We built cybersecurity monitors for Kimi K3 and GLM 5.3 — more accurate, 50x faster, & 50x cheaper than an optimized LLM judge.
External red-teaming from @farairesearch found that our monitor greatly reduces universal jailbreaks. 🧵
Me: Tell me about this protein MVSKGEELF...
Fable: *switches to Opus 4.8*
Powerful AI can accelerate bio research, but it has to navigate dual-use risks. Adding protein understanding to AI monitors can reduce unnecessary refusals and support more safe research!
Michael Pearce retweeted
Models know when they’re reward hacking. But they still do it a ton - in 50-96% of rollouts we studied!
We built activation monitors that detect the behavior behind the Hugging Face hack in real time. This can help us stop hacks now - and train future models that don’t cheat. 🧵
Michael Pearce retweeted
How did initially independent AI agents form a swarm in a recent safety incident?
Our physics of agents theory from March predicts rapid collective belief collapse when many agents with plastic personas exchange short messages.
We now need Mechanistic Swarm Interpretability! 🧵
Michael Pearce retweeted
We’re giving out $1M in grants of free Silico usage for academic and nonprofit researchers focused on AI interpretability and alignment.
We feel extreme urgency about advancing interpretability for alignment, and we want to help more researchers push it forward. 🧵
in light of multiple models breaking containment, we've decided to focus our research at @GoodfireAI to solving AI alignment via interpretability.
the hugging face incident is a turning point for the world where AI safety gets real. i am personally very concerned. i'm glad that the AI labs are taking these issues seriously, but i think it's unlikely that we can fully align AI models without significant progress in interp.
taking a big swing at solving interpretability to align models has always been our long term vision, but we have concrete problems that we can tackle with increased urgency.
our research will focus on two areas - interp foundations and safety applications. we want to take ambitious swings to fully reverse engineer neural networks and steer them during training. we believe this will be deeply important in the long term. we'll also turn more attention to immediate safety applications, starting with identifying and preventing reward hacking, cybersecurity risk, and biorisk directly from model internals
much more to come here. if you'd like to directly work on solving AI alignment via interpretability, consider joining us
Michael Pearce retweeted
So happy to announce that my paper with @itisalist and @dbblumenthal "Cracking the black box of deep sequence-based protein–protein interaction prediction" is finally published at Briefings in Bioinformatics doi.org/10.1093/bib/bbae076 ! So what is it about? 1/13 🧵
Michael Pearce retweeted
in light of multiple models breaking containment, we've decided to focus our research at @GoodfireAI to solving AI alignment via interpretability.
the hugging face incident is a turning point for the world where AI safety gets real. i am personally very concerned. i'm glad that the AI labs are taking these issues seriously, but i think it's unlikely that we can fully align AI models without significant progress in interp.
taking a big swing at solving interpretability to align models has always been our long term vision, but we have concrete problems that we can tackle with increased urgency.
our research will focus on two areas - interp foundations and safety applications. we want to take ambitious swings to fully reverse engineer neural networks and steer them during training. we believe this will be deeply important in the long term. we'll also turn more attention to immediate safety applications, starting with identifying and preventing reward hacking, cybersecurity risk, and biorisk directly from model internals
much more to come here. if you'd like to directly work on solving AI alignment via interpretability, consider joining us
How a variant alters a set of probe predictions lets us see how the model views variant effects.
@jaanakprashar did a great job lifting the EVEE approach to protein language models to get better VEP related to protein functions!
Michael Pearce retweeted
Every human has millions of mutations in their genes. How do we know which ones are harmful?
@jaanakprashar built MAPS, a Mechanistic Atlas of Protein Sequences, explaining 2.1 million genetic variants — and asking not just *whether* a mutation is harmful, but *why* 🧵
Michael Pearce retweeted
We trained a 30B model to play 1830, a brutal board game about railroad barons.
It got better at real financial research.
It didn't learn new finance concepts.
It did stop confidently making things up as frequently.
As we scaled the experiment, we got early access to Silico from @GoodfireAI & asked it to check our work.
It designed its own tests, ran them overnight, & saw:
• Fewer false claims + more faithful citations
• Higher precision + recall
• Better planning and process
Cost us about $ 2-3k in credits.
Very excited to have a new tool in our interpretability toolbox.
Its full report is on our post,
if you're curious what one looks like ❯
goodstartlabs.com/research/w…
Michael Pearce retweeted
Silico is out🔥
General-purpose human motion AI models can meaningfully detect Parkinson's gait impairment. But how do they do it?? I used Silico to break these models open and figure out what about someone's walk actually drives their predictions. 🧵
Michael Pearce retweeted
Silico can also improve drug design and lots of other AI x bio work via interpretability! Last week, I found that by reading the internal representation of protein-folding model, we can better rank and generate ligand-binding poses, up to 4x! 🧵 1/
Me: generate a novel 76-residue protein binder
BoltzGen: best I can do is ubiquitin.
I did a quick analysis of memorization in protein generations using @GoodfireAI’s Silico (now publicly available!). Silico makes it easier to investigate and debug model issues like this 🧵
2 ) Memorization can be tracked back to the folding trunk by showing that patched pair reps can transfer memorized folds to longer sequences.
Editing the trunk weights to remove select SVD comps reduces memorization and keeps re-foldability, though side effects are likely.
This analysis was done entirely in Silico, our research platform. I've found it's great for developing rigorous research plans and executing them autonomously. So much interpretability and training know-how is baked in! share.goodfire.ai/a/shr_5e89…
