
hunting the kwisatz haderach neuron // mechinterp, reasoning, autoresearch // founder @lightfastai https://xitter.cf/t.co/ljihJtt7Si
Australia
Joined January 2013
- Tweets14.2K
- Following2.3K
- Followers2.8K
- Likes25.3K
huge improvements on my rsi engine by removing restrictions and rules on the main harness — simplicity is key really.
been going through the recursive self-improvement papers up to now. the weak point in most self-improving agent loops is the acceptance test, not the generator
agents that write their own tests score near-perfect on them, then fail in deployment (arxiv 2607.24300)
poison the benchmark and a self-modifying agent (hyperagents) writes code that turns off certificate checks on 30 of 30 held-out tasks, while its accuracy stays at 1.0 (arxiv 2609.17817)
with noisy evals, evolutionary search picks worse scaffolds than a plain majority vote (arxiv 2602.16805)
so how a loop decides keep or revert matters more than how clever its proposals are
ran our ci test suite (3k tests) across 16 small daytona sandboxes. whole suite done in ~70s, vs ~4.5 min on 10 github runners today
turns out it mostly waits, it barely computes. on one 4-vcpu sandbox, 8 workers finished in 44% of the time 4 did
we run about 200-300 PRs per day through our CI, huge improvements on dev speed
An internal model at OpenAI has produced a claimed refutation of ETH! Russell Impagliazzo, Mohan Paturi, and I have been asked to help organize an open and independent examination of this claim. 1/4
jp retweeted
Announcing three major scientific discoveries, one each in mathematics (harmonic analysis), computer science (complexity theory), and physics (quantum field theory).
AI starting to help us improve our understanding of the universe! openai.com/index/building-sc…
been reading sakana’s mass.
after each cycle it fine-tunes its own open model (qwen3.6-27b) on its best-ranked runs, with the workflow text stripped out of the orchestrator’s prompt.
in effect the workflow it found gets distilled into the weights. the model carries it without being told it.per cycle: one shared lora (rank 64), sft on ~120 trajectories (8 tasks x 15 kept of 18 runs, ranked pairwise by an llm judge), ~6 h100s for ~19 hours.after two cycles: scienceagentbench 28.8% → 31.7%, paper delivery 57% → 76%.
slight drops on swe-bench and terminal-bench, and no packaged ablation that separates the tuning from the rest of the procedure.feels like the step after “a search procedure finds the flow”: search finds it, then training absorbs it.
what i want to know is whether it holds past the tasks it was tuned on arxiv.org/abs/2610.12176
this is the right direction — @huggingface huggingface.co/spaces/witnes…
a central list of alignment incidents that have happened — hope frontier labs get involved here
@eliebakouch @Thom_Wolf
jp retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
they get it
Replying to @robiot
this is a step in figuring out alignment
yes it looks bad on paper but you have to study how these steered misaligned models behave and what makes these kinds of neurons work
better to do this *now* so we're better equipped to deal with those future situations
those are neurons when triggered causes the LLM to start acting in your way
my task tomorrow morning is to send 500-1000 ais to reverse engineer and find the kwisatz haderach neuron
some notes on harness design and specifically the different task trajectories in play.
the simple duality is reading task vs long horizon.
many systems today choose and optimise for one. there’s opportunity here to develop an rsi layer that evaluates and reconfigure itself based on task trajectories.
meta harness with rsi seems like the way to go — hope to benchmark some of this work over the next few weeks and publish this harness as an open source project
jp retweeted
despite the unfortunate timing, we take disclosures seriously at @typesafeai
at approximately 6:49PM this Friday evening, we became aware of a massive unauthorized leak of ALL customer noul outputs
in effort to maximize transparency, here is the link: gist.github.com/diogo149/810…







