Exploring the interconnectedness of all things. mech interp systems @GoodfireAI 🏳️⚧️
Mapping the Rhizome
Joined June 2017
- Tweets16.8K
- Following2.4K
- Followers6.8K
- Likes36.1K
I turned qwen into a catgirl
Inspired by single neuron steering experiments by @_lyraaaa_, @Sauers_ and @jfonsecarivera, I tried to do something new:
Given a (partial) neuron atlas, can we compose a set of known neurons to get a desired output persona?
Turns out we can!
Composing the neurons for mascot, cat-word bias, kawaii, and female causes qwen to identify as a catgirl named Meecot.
A cool example to support the idea that native neurons could actually be useful, understandable building blocks.
I will literally die if I don't get tickets to this
PORTER ROBINSON b2b NINAJIRACHI
san francisco 11.28.2026 【= T𖥦T =】
w @dylanbrady @dossxoxo @ninalasvegas @clear__cast
Long post about my thoughts on this:
I'm so excited that this is finally out! Optimizing this until scaling it out is actually feasible and building the systems to perform the scaling has been my job for the last couple of months. This work is some of the most deserving of my time and energy that I can imagine.
There's been a lot of Discourse in the interpretability community lately about the neuronal and featurizer paradigms (not enough discussion of the causal abstraction paradigm imo..), the LRH and superposition, etc but no matter what paradigm we find ourselves in (and remember they can co-exist and describe different phenomena!) it's paramount that we accelerate the science of how models handle a wide array of tasks, how compositionality is constructed through intersection and containment of internal machinery (whether neurons, circuits, subspaces, or whatever else). Our current approaches have been too isolated to specific task and model combinations, but now that we have proven methodologies we can begin the process of "normal science" by applying them to population studies.
Once we've proven this out, it will be time to scale the effort beyond even the model-specific setting. This is where things get really interesting! Given how many different axes models vary on, we need to see what findings hold across which axes (model size, architecture, attention schemes, etc) if we are ever to develop "laws" of interpretability. Only then can we start to apply mechanistic alignment techniques broadly across the industry and have a reasonable expectation that it will work.
Personally, I'm quite optimistic based on our early results. I firmly believe that this is a solvable problem, but it won't be easy and requires an enormous amount of domain expertise (I'm so grateful to work with such talented researchers) to keep the agents from slopping it up. It will require an even more enormous about of resources, which is why it has to be an effort of the community at large. When we're ready, I hope that we can rally everyone who has the drive to solve these hard problems to use our encyclopedia and to help us fill it out.
if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
I'm becoming much more of a radical Rustacean than I used to be. It's been my favorite language for a while, but the flood of user after free CVEs in code that is very widely deployed is so insane that I am starting to believe it's irresponsible to use any other systems language.
Absolute insanity
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing-ou…
Nymph retweeted
Evergreen
Presenting my grand unified theory of ML researcher impact: Your impact is directly proportional to how much pain you cause to infra.
Fundamentally, you can only inflict pain upon infra if your approach actually works. And the better your approach works the more pain infra is forced to endure.
So, to give some examples:
- MoE's add a ton of data-dependent computation => pain (shazeer++)
- GDN/KDA are the most complex architecture I've been forced to care about and a very annoying matrix inversion => pain (sonta++)
- Muon is much more annoying than Adam and causes annoying restrictions on parallelism => pain (keller/jeremy++)
- RL scaling forced many researchers to care about LLM inference and RL infra as a category => pain (tworek++)
Even papers like Attention Is All You Need have lead to significant pain! Before transformers were invented everyone was running small jobs and I never needed to think about kv-caches or 6D parallelism.
The blog post with visual explainers is insanely good, one of the best put together technical articles I've ever seen
We’re sharing Dist-MoE, open-source library from Meta SuperIntelligence Labs (MSL @AIatMeta ). Dist-MoE provides fused communication and compute kernels for BF16/MXFP8 training and inference, plus NVFP4 inference.
✅ Fused communication and compute kernels
✅ Full CUDA graph support
✅ Dynamic activation packing across layers into a bounded buffer
✅ Dynamic, capacity-driven recompute
✅ Forward and backward kernels for BF16/MXFP8
✅ No shared-expert branch required
Some highlights from our GB300 NVL72 benchmarks:
• Up to 1.40× speedup over Mixture-of-Kittens (MoK) in our imbalanced, ten-layer MXFP8 training workload across 64 GPUs. Dist-MoE packed activation state across all ten layers with zero capacity-induced recompute.
• 3.09 useful PFLOP/s per GPU (3,090 TFLOP/s) for NVFP4 Mega inference across 16 GPUs.
• 1.86 useful PFLOP/s per GPU (1,856 TFLOP/s) for MXFP8 Staged training across 64 GPUs.
• 1.27 useful PFLOP/s per GPU (1,271 TFLOP/s) for BF16 training across 64 GPUs.
• Sub-linear scaling under imbalance: in NVFP4 inference across 16 GPUs, 8× the received rows on the busiest GPU required 4.82× the iteration time; about 40% lower iteration time per received row.
Full configurations, raw results, and measurement definitions are in the blog.
@levendlee and I also put together an interactive visual explainer covering token dispatch and combine, quantization, forward/backward kernels, activation packing, dynamic recompute, and memory planning.
Blog:
meta-pytorch.org/dist_moe/pr…
Repository:
github.com/meta-pytorch/dist…
TorchTitan (@PyTorch ) integration:
github.com/pytorch/torchtita…
If you work on MoE training, inference, or GPU kernels, take a look. We welcome feedback and contributions.
#PyTorch #MixtureOfExperts #MoE #OpenSource #TorchTitan #DistMoE
This is a great thread. More people should be aware of how much the differences between theory and reality matter in neural nets. The things we do to make it all fast enough cause small differences that propagate out enough to be learned on.
I got honorable mention :3 I'm finally wrapping up the last of the changes I wanted to make before a writeup on Crosstalk too, I think I'll be posting something about it this weekend!
God I'm so proud to work here. Safety and medicine are, imo, the two most important applications of interpretability imo, preventing the bad outcomes and pushing for the good ones. We're making strides in both.
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it successfully reclassified as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!