hunting the kwisatz haderach neuron // mechinterp, reasoning, autoresearch // founder @lightfastai https://xitter.cf/t.co/ljihJtt7Si

Australia
Joined January 2013
huge improvements on my rsi engine by removing restrictions and rules on the main harness — simplicity is key really.
34
been going through the recursive self-improvement papers up to now. the weak point in most self-improving agent loops is the acceptance test, not the generator agents that write their own tests score near-perfect on them, then fail in deployment (arxiv 2607.24300) poison the benchmark and a self-modifying agent (hyperagents) writes code that turns off certificate checks on 30 of 30 held-out tasks, while its accuracy stays at 1.0 (arxiv 2609.17817) with noisy evals, evolutionary search picks worse scaffolds than a plain majority vote (arxiv 2602.16805) so how a loop decides keep or revert matters more than how clever its proposals are
63
ran our ci test suite (3k tests) across 16 small daytona sandboxes. whole suite done in ~70s, vs ~4.5 min on 10 github runners today turns out it mostly waits, it barely computes. on one 4-vcpu sandbox, 8 workers finished in 44% of the time 4 did we run about 200-300 PRs per day through our CI, huge improvements on dev speed
1
3
133
An internal model at OpenAI has produced a claimed refutation of ETH! Russell Impagliazzo, Mohan Paturi, and I have been asked to help organize an open and independent examination of this claim. 1/4
294
745
526
7,235
3,579,914
Announcing three major scientific discoveries, one each in mathematics (harmonic analysis), computer science (complexity theory), and physics (quantum field theory). AI starting to help us improve our understanding of the universe! openai.com/index/building-sc…
665
2,026
851
16,029
3,073,623
been reading sakana’s mass. after each cycle it fine-tunes its own open model (qwen3.6-27b) on its best-ranked runs, with the workflow text stripped out of the orchestrator’s prompt. in effect the workflow it found gets distilled into the weights. the model carries it without being told it.per cycle: one shared lora (rank 64), sft on ~120 trajectories (8 tasks x 15 kept of 18 runs, ranked pairwise by an llm judge), ~6 h100s for ~19 hours.after two cycles: scienceagentbench 28.8% → 31.7%, paper delivery 57% → 76%. slight drops on swe-bench and terminal-bench, and no packaged ablation that separates the tuning from the rest of the procedure.feels like the step after “a search procedure finds the flow”: search finds it, then training absorbs it. what i want to know is whether it holds past the tasks it was tuned on arxiv.org/abs/2610.12176
1
3
153
this is the right direction — @huggingface huggingface.co/spaces/witnes… a central list of alignment incidents that have happened — hope frontier labs get involved here @eliebakouch @Thom_Wolf
78
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
886
2,634
1,444
19,854
4,565,599
lets find a new one shall we
Replying to @_lyraaaa_
How did you find this?
113
320
91
6,772
296,857
been on a deep build of rsi repo for openkernelchallenge — almost done. looking to climb soon.
1
1
156
they get it
Replying to @robiot
this is a step in figuring out alignment yes it looks bad on paper but you have to study how these steered misaligned models behave and what makes these kinds of neurons work better to do this *now* so we're better equipped to deal with those future situations
3
149
good morning — back on the hunt for the kwisatz haderach neuron.
1
76
those are neurons when triggered causes the LLM to start acting in your way my task tomorrow morning is to send 500-1000 ais to reverse engineer and find the kwisatz haderach neuron
if you have a qwen3.5 4b around, please try steering neurons L14 N8949 +1 and L13 N7956 -1 now you have evil qwen recipe: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖` at the down_proj at every input
1
1
145
wadafaq
if you have a qwen3.5 4b around, please try steering neurons L14 N8949 +1 and L13 N7956 -1 now you have evil qwen recipe: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖` at the down_proj at every input
97
some notes on harness design and specifically the different task trajectories in play. the simple duality is reading task vs long horizon. many systems today choose and optimise for one. there’s opportunity here to develop an rsi layer that evaluates and reconfigure itself based on task trajectories. meta harness with rsi seems like the way to go — hope to benchmark some of this work over the next few weeks and publish this harness as an open source project
92
despite the unfortunate timing, we take disclosures seriously at @typesafeai at approximately 6:49PM this Friday evening, we became aware of a massive unauthorized leak of ALL customer noul outputs in effort to maximize transparency, here is the link: gist.github.com/diogo149/810…
449
315
260
6,159
1,193,220