Michael 🔸

I was asked by a journalist to comment on this piece. The comments I wrote up (on Friday evening) are included below, so that I have a public place where my view is fully represented. 1. I appreciate the critical reassessment of some of the existing arguments in favor of superposition. It does look like alternative explanations may be viable. 2. I should say that, despite being an author on perhaps 40 papers on interpretability, I have never really worked on the superposition hypothesis per se. There's a lot more to interpretability than this one idea! We did, though, develop a benchmark (RAVEL) and associated metrics for assessing the extent to which features in neural networks can be disentangled: aclanthology.org/2024.acl-lo… We found that many neurons are polysemantic, but we made no claims about the more specific notion of superposition. It would be great to see RAVEL results for newer models. 3. A fundamental motivation for interpretability research is preparing for an uncertain future full of strange new models with surprising new capabilities. Superposition is a technical insight about how neural networks can learn to pack unrelated features into a single neuron. Even if present-day models rarely do this, we need to be ready for future models. Cammarata's analysis seems to trace legibility in part to architectural choices. Future architectures might be very different. If interpretability researchers don't prepare us for that, who will? 4. Present-day models may make use of superposition as well! Cammarata suspects that they don't, or do this so rarely that we could even ignore the actual instances ("doesn't matter for study"). I would characterize this as a hunch based on a lot of initial heuristic empirical work. In the normal course of scientific research, this would be the start of a process of figuring out precisely how to rigorously test this as a hypothesis. 5. For example, the piece has a nice figure that Cammarata characterizes as indicating that a particular neuron has positive activation on pointed things and negative activation on round things. Cool! Let's define some independent criteria for those classifications so that we can test this objectively. (It is possible that the paper Cammarata is preparing will include these evaluations.) 6. The piece expresses the intuition that interpretability is easier, in some sense, with better models. This intuition has been very common among researchers for years. At the same time, I understand why @JacobHHilton would say the opposite: teapaw.org/JacobHHilton/status/21… 7. The piece seems to allude to a technical proposal, but it does not give the details. As far as I can tell, the ideas have substantial overlap with this work (led by two former students of mine): Arora, Wu, et al. 2026 (arxiv.org/abs/2601.22594). Cammarata does cite this. I am guessing there could real differences that will be interesting to see. I would hope that Cammarata explains the differences and reports comparable evals to Arora et al, at least for language models. I appreciate that he looked at "thousands" of examples (the essay says this a lot), but even a large number here is not a substitute for rigorous evaluation. 8. In my research group at Stanford, we are working on lots of very hard problems in interpretability. As far as I can tell, nothing about this blog post makes these problems any easier, and certainly none of them are solved by this work. It would not have occurred to me to say this at all, since it is not even reasonable to have this expectation, but very prominent people on social media seem to believe that it is an all-encompassing breakthrough. Some immediately extrapolated to an optimistic story about AI safety that is not supported by anything in this piece. (Highly interpretable agents could of course do dangerous things.) 9. It must be said that Cammarata's framing did encourage these reactions. The header of the post page says, "An informal research essay and working hypothesis about neural networks", which suggests something quite preliminary and tentative. The piece also says, "Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we've assumed". However, his announcement tweet (teapaw.org/nickcammarata/status/2…) reworked this considerably: "I now think ambitious interpretability will be solved by the time we need it most". Strictly speaking, this commits to very little beyond optimism, but it is written to invite many inferences.