I think it is a fundamental conflict in standard transformers that although its value matrices are "looking for" token embeddings as inputs, token embeddings end up having minimal contributions. We can clearly see that value matrices are looking for token embeddings, because among all layer outputs that can go into the residual stream, only the token embedding consistently project a large portion of its norm onto top input dimensions of value matrices. In fact, if we compare how much the token embedding vs the complete residual stream projects itself onto value matrices, token embeddings across heads and layers almost always have a larger projection. However, we can also see that token embeddings are not contributing much to value vectors, if at all. If we compute the squared norm of the output of value matrices giving only token embedding vs other layer outputs, we find that token embedding has the most negligible contribution. This creates an intrigue -- are there architectural constraints pulling embeddings back in the hidden space? We believe so, from various places -- such as the fact that embedding is the only component in the residual stream that is layer-normed, before adding to the stream. Ongoing work. Blog: riddlehe.github.io/blog/impo…

Interested? Check out this paper from UMD grads at Apple:

if you have a qwen3.5 4b around, please try steering neurons L14 N8949 +1 and L13 N7956 -1 now you have evil qwen recipe: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖` at the down_proj at every input

Dropped a 58 min visual explanation of this essay on YouTube No pre requisites except basics of deep learning... as usual we will learn it from first principles

AI 时代，年轻人应该怎么办？很简单，几个关键词： 1. Stranford CS336 2. Sebastian Raschka: Build a Reasoning Model 3. Nathan Lambert: Reinforcement Learning from Human Feedback 4. 谭成义三分化

The complete reasoning-from-scratch video series is now up (and linked) here: magazine.sebastianraschka.co… (Taking a short break but will be back with LLM model and research updates in November!)

"Start Doing Graph Theory" is a free, 478-page textbook (October 2026) that provides a comprehensive introduction to graph theory, combining mathematical rigor with a problem-solving approach. The book covers graph isomorphisms, paths and cycles, trees, directed graphs, Eulerian and Hamiltonian cycles, matchings, graph coloring, planar graphs, network flows, and much more. It is a valuable resource for university students and anyone interested in developing a deeper understanding of graph theory and its applications. The book also includes numerous exercises that help readers put theoretical concepts into practice. Definitely worth adding to your mathematics library. vertex.degree/pdf/book.pdf

🛠️ [𝐏𝐫𝐢𝐦𝐞𝐫] 𝐇𝐚𝐫𝐧𝐞𝐬𝐬 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 • harness.aman.ai - I’ve increasingly found that improving an AI agent isn’t only about improving the underlying model. The harness around it, including tools, memory, context, verification, and control flow, can materially change what the system can accomplish. - I find self-improving harnesses especially exciting: agents can inspect execution traces, diagnose recurring failures, modify their surrounding system, and evaluate whether those changes actually improve performance. This primer covers harness design and self-improvement end-to-end, from autoresearch and meta-harness to memory, evaluation, safety, and scaling. 🔹 𝐇𝐚𝐫𝐧𝐞𝐬𝐬 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 • Why the Harness is a First-Class Capability Variable • Autoresearch: Agent-Run Optimization Loops by @karpathy • Meta-Harness: Optimizing the System Around a Fixed Model 🔹 𝐒𝐞𝐥𝐟-𝐈𝐦𝐩𝐫𝐨𝐯𝐢𝐧𝐠 𝐇𝐚𝐫𝐧𝐞𝐬𝐬𝐞𝐬 • Trace-Driven Harness Self-Modification ("Self-Harness: Harnesses That Improve Themselves") • Open-Ended Agent Evolution ("Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents" by @jennyzhangzt et al.) • Learning How to Improve the Harness ("Harness Learning Enables Generalizable Test-Time Adaptation" by @ZAlvin39105 et al.) 🔹 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞, 𝐌𝐞𝐦𝐨𝐫𝐲, 𝐚𝐧𝐝 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 • Proposer Agents, Evaluators, and Validation Gates • Durable Memory and Execution Traces • Context Compaction and Retrieval 🔹 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧, 𝐒𝐚𝐟𝐞𝐭𝐲, 𝐚𝐧𝐝 𝐒𝐜𝐚𝐥𝐢𝐧𝐠 • Evaluate the Model + Harness As a Compound System • Regression Testing and Trace-Driven Failure Diagnosis • Sandboxed Self-Modification and Immutable Evaluation • Scaling Search While Preserving Attribution Primer written in collaboration with @VinijaJain. #ArtificialIntelligence #GenAI #Harness

Probability paths describe how a distribution evolves from source to target. The Linear Path connects each paired sample with a straight line. Alternative paths behave differently. This Variance-Preserving Path uses cosine–sine interpolation to trace a curved, elliptical arc.

Models know when they’re reward hacking. But they still do it a ton - in 50-96% of rollouts we studied! We built activation monitors that detect the behavior behind the Hugging Face hack in real time. This can help us stop hacks now - and train future models that don’t cheat. 🧵

Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:

1/ JEPA selects which features its world model keeps. But how? Will it keep a feature it can't predict, like a traffic light? Judging each feature alone, earlier work says no. We show evidence that this is wrong and introduce a new characterization that explains why. 👇🧵

Quite happy to say that our paper on representation learning for dynamical systems was accepted to #NeurIPS2026 and I'll be presenting it in Sydney this December! Very succinctly, our work, titled "Sparse Koopman Autoencoders Identify Local Dynamical Regimes in Multibasin Systems", is about learning linear representations of state for dynamical systems with multiple basins of attraction. Inducing sparsity in a latent representation can be a very powerful method to model multiple basins of attraction with linear dynamics. By training a Koopman autoencoder with a bias for producing sparse latent representations, we find that a sparse-latent model both forecasts the dynamics better and has greater ability to identify the current local regime of the current state compared to a model without sparsity-inducing effects. More broadly, I think our work on sparse Koopman autoencoders could be relevant to both the robotics/world modeling and interpretability communities (just brainstorming here). - Robotics: To my knowledge, switching dynamics are common in robotic systems, i.e. trajectories seem to transition between dynamical modes. With a sparse code, you could get a switch between modes for free. - Interpretability: Sparse autoencoders might not be all the rage anymore, but I think Koopman modeling is still very natural for understanding the dynamics of models that apply the same computation at multiple steps to advance the dynamics. This applies to most recurrent models, such as looped/weight-tied transformers (dm me if you want to discuss!) Thanks to my amazing collaborators @ukreddy126 @mahanfathi @apsarathchandar @RGoroshin. I was thinking about what to say about this (clueless grad student tries to figure out social media) and this post ended up being really late lol, my bad. See you in Sydney! Paper: arxiv.org/abs/2608.29057 Code: github.com/aidanmrli/skae

I think there are some issues in the recent “Already Clean” essay, and I’m concerned it may have incorrectly updated people on important safety-relevant questions. My thoughts (including some reproductions): livgorton.com/already-clean-…

We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected! This old baseline deserves a second look. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory. We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary! More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions. Try it: pip install icalens pypi.org/project/icalens/ Paper: arxiv.org/abs/2606.11722v2 1/8

Tested this! I flattened every neuron's weakest activations Overall loss barely changes at first but the models get REAL bad at memorized text. So the weak tail does, as Neel expected, contain memorized things. Weaker activations seem to correspond to less generalizable content

There has been much discussion recently about how interpretable neurons in LLMs are. Today we're announcing our paper from January's cohort of Anthropic Fellows, where we show neurons may be a stronger basis for interpretability than commonly assumed. 🧵 (1/6)

Why? How did we ever get so lucky find meaningful neurons inside neural networks? We finally have a theory: arxiv.org/abs/2610.20851 🧵🧵🧵

This is interesting work, but I don't think it's a significant update to the big picture of interpretability. I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc. I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point. I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes! Similarly, my impression is that SAEs generally don't find things that are a single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework. And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.

I was asked by a journalist to comment on this piece. The comments I wrote up (on Friday evening) are included below, so that I have a public place where my view is fully represented. 1. I appreciate the critical reassessment of some of the existing arguments in favor of superposition. It does look like alternative explanations may be viable. 2. I should say that, despite being an author on perhaps 40 papers on interpretability, I have never really worked on the superposition hypothesis per se. There's a lot more to interpretability than this one idea! We did, though, develop a benchmark (RAVEL) and associated metrics for assessing the extent to which features in neural networks can be disentangled: aclanthology.org/2024.acl-lo… We found that many neurons are polysemantic, but we made no claims about the more specific notion of superposition. It would be great to see RAVEL results for newer models. 3. A fundamental motivation for interpretability research is preparing for an uncertain future full of strange new models with surprising new capabilities. Superposition is a technical insight about how neural networks can learn to pack unrelated features into a single neuron. Even if present-day models rarely do this, we need to be ready for future models. Cammarata's analysis seems to trace legibility in part to architectural choices. Future architectures might be very different. If interpretability researchers don't prepare us for that, who will? 4. Present-day models may make use of superposition as well! Cammarata suspects that they don't, or do this so rarely that we could even ignore the actual instances ("doesn't matter for study"). I would characterize this as a hunch based on a lot of initial heuristic empirical work. In the normal course of scientific research, this would be the start of a process of figuring out precisely how to rigorously test this as a hypothesis. 5. For example, the piece has a nice figure that Cammarata characterizes as indicating that a particular neuron has positive activation on pointed things and negative activation on round things. Cool! Let's define some independent criteria for those classifications so that we can test this objectively. (It is possible that the paper Cammarata is preparing will include these evaluations.) 6. The piece expresses the intuition that interpretability is easier, in some sense, with better models. This intuition has been very common among researchers for years. At the same time, I understand why @JacobHHilton would say the opposite: xitter.cf/JacobHHilton/status/21… 7. The piece seems to allude to a technical proposal, but it does not give the details. As far as I can tell, the ideas have substantial overlap with this work (led by two former students of mine): Arora, Wu, et al. 2026 (arxiv.org/abs/2601.22594). Cammarata does cite this. I am guessing there could real differences that will be interesting to see. I would hope that Cammarata explains the differences and reports comparable evals to Arora et al, at least for language models. I appreciate that he looked at "thousands" of examples (the essay says this a lot), but even a large number here is not a substitute for rigorous evaluation. 8. In my research group at Stanford, we are working on lots of very hard problems in interpretability. As far as I can tell, nothing about this blog post makes these problems any easier, and certainly none of them are solved by this work. It would not have occurred to me to say this at all, since it is not even reasonable to have this expectation, but very prominent people on social media seem to believe that it is an all-encompassing breakthrough. Some immediately extrapolated to an optimistic story about AI safety that is not supported by anything in this piece. (Highly interpretable agents could of course do dangerous things.) 9. It must be said that Cammarata's framing did encourage these reactions. The header of the post page says, "An informal research essay and working hypothesis about neural networks", which suggests something quite preliminary and tentative. The piece also says, "Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we've assumed". However, his announcement tweet (xitter.cf/nickcammarata/status/2…) reworked this considerably: "I now think ambitious interpretability will be solved by the time we need it most". Strictly speaking, this commits to very little beyond optimism, but it is written to invite many inferences.