What a lame cult 🥱
Anthropic Researcher Tao Lin: Pretty much all humans on earth will become permanently unemployable because of AI
"It's very likely that pretty much all humans on earth will become permanently unemployable"
"And will need to strictly rely on government welfare to sustain themselves"
"Human employment will go away."
1
16
The average MD graduate has a similar number of citations, due to the nature of publishing in the health sciences, where meta analyses and systematic reviews are common.
This is nuts. Abdul El-Sayed’s h-index on Google Scholar is 37 — higher than that of your average tenured professor at an Ivy League school.
1
76
alireza retweeted
1/ JEPA selects which features its world model keeps.
But how?
Will it keep a feature it can't predict, like a traffic light?
Judging each feature alone, earlier work says no.
We show evidence that this is wrong and introduce a new characterization that explains why. 👇🧵
alireza retweeted
Want to understand LLMs? Start with how neural networks store memories.
This idea deeply shapes how I think about them, so I'm glad to see @heyyalexwang explaining it this clearly, and to see it getting the attention it deserves.
Using this occasion to share some of the recent research I’ve been part of on using this perspective to better understand LLMs:
1. LLMs are extremely good at storing knowledge, but how do they learn to do so? We study this question in arxiv.org/abs/2503.21676, showing that attention circuits need to be created before models can store memories. These attention circuits emerge during training; we study how this depends on data in arxiv.org/abs/2505.17863.
2. The memory setup in the blog, and the standard one more generally, is to have only one possible value, e.g. a next token. In practice, LLMs have to model different possible future tokens; we show in arxiv.org/abs/2608.15448 that such “ambiguity” requires larger networks to store!
3. What happens when a network with a lot of knowledge learns new memories: it catastrophically forgets. This is a nice toy setup to systematically study it and there are some surprising other phenomena that one can study. In arxiv.org/abs/2610.08718 we study spurious forgetting: in some cases there is forgetting in the sense that memories cannot be recalled, but there are still present if probing the network in the right way.
4. One can see post-training as a slight modifications of the memory of the network. We use this perspective in arxiv.org/abs/2610.03529 to give some intuition about how different distillation strategies affect model behavior; this is imo a cool setup to understand some distillation behaviors.
Bonus. From the arguments in the quoted blog, one can heuristically derive scaling laws. I gave a tutorial on that (which reviews some of the work linked in the blog) nicolaszucchet.github.io/Tut…; there is even a colab where you can train your own memories and derive their scaling laws!
There are still many open questions related to memories, knowledge and LLMs, and I hope to see a lot more of cool research to do in this direction!
alireza retweeted
Interesting food for thought, and I hope people will take interpreting MLP neurons more seriously. But “recursive generalization” vs. superposition seems to come down to whether the concepts sharing a neuron are related, and if the method is “look harder until the activations share a story,” you’ll always find one. I agree with @neelnanda that I would classify a fair amount of the examples in the post as polysemantic.
I do, however, think the essay’s proposal to understand how neurons branch out throughout training could be fruitful. But as an interpretability community, we really need to go beyond vibes and be rigorous about developing evaluations as well as toy models and theory that are predictive of real phenomena.
Due to the revived interest in interpreting neurons, I wanted to highlight some papers relevant to this discussion following @aryaman2020's post:
@davidbau et al pnas.org/doi/abs/10.1073/pna… object detector neurons in CNNs
@AlecRad et al. arxiv.org/abs/1704.01444 sentiment neuron
@YGandelsman et al arxiv.org/pdf/2406.04341 text-based interpretation of neurons in CLIP
@alesstolfo et al arxiv.org/pdf/2406.16254 neurons that modulate confidence in language models
@TamarRottShaham @cogconfluence et al. arxiv.org/pdf/2404.14394 automated interpretability of vision neurons through agents
me @YGandelsman arxiv.org/abs/2306.09346 and arxiv.org/abs/2606.03990 universal "Rosetta neurons" and how they scale with model size
@YizhouLiu0 et al. arxiv.org/pdf/2505.10465 how superposition can affect neural scaling laws
@aryaman2020 @ZhengxuanZenWu et al arxiv.org/pdf/2601.22594 circuit tracing in the neuron basis
@wesg52 et al. arxiv.org/pdf/2305.01610 finding interpretable neurons with sparse probes
@gabeeegoooh et al distill.pub/2021/multimodal-… CLIP neurons responding to multimodal concepts.
among many others
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
alireza retweeted
The human brain is approaching its ChatGPT moment.
I spent 6 months writing Phase Lock: a manifesto on how brain computer interfaces can help align AI and preserve human agency.
If you only ever read one thing from me, read this.
Wiener’s cybernetic cat once said, “The best material model of a cat is another, or preferably the same, cat.” In other words, the G in AGI is nonsense.
3
118
Can someone explain why Iranian students are reportedly being rejected from graduate programs in many countries, e.g. Germany for 2027 even in non sensitive fields? I thought the distinction between ordinary Iranian citizens and the regime was clear.
alireza retweeted
🚨🚨🚨A research project idea! How to measure world models?
Everyone's talking about world models these days. World model here, world model there. We can argue about what "world model" actually means, and we have some interesting results on that, but let's assume some hand-wavy definition for now. The real question is how you tell a good world model from a bad one.
Here's my bet on what actually matters, and it isn't how real the video looks or whether you can read off physical quantities from some giant latent. It's the structure of how the latents evolve. Say a rock is flying towards you and you want to plan an escape — do you care about its velocity? its energy? the projection of its velocity onto the 2D plane? They're all nonlinearly related, so chasing any one of them is the wrong target. What matters more is whether there's a simple, interpretable rule for how the latents move. And it has to be the latents — the real degrees of freedom aren't the pixels. Write down the map from past latents to future latents: what's its Jacobian? The world is noisy, so do errors propagate in a reasonably uniform way, or are there points where a tiny error blows up immediately? And separately, find the smallest latent space that still works and check whether its dimension matches what physics says it should be.
Meanwhile the field measures world models in a dozen ways, and they quietly disagree with each other. Representational probing puts a linear probe on the representation and asks what's decodable. Information-based methods try to measure something like the predictive information the latents carry about the future. Rollout error runs the model forward and tracks how fast it drifts from reality. Then there's downstream task success — does it help on some task — versus closed-loop utility, where an agent actually plans with it and you see whether it succeeds. Different things, and they often disagree.
So two ways to make sense of the mess. One is empirical: take many settings and many models, measure all these metrics, and map how they correlate, similar to the "Fantastic Generalization Measures and Where to Find Them," paper, but for world models. The other is to stop treating them as separate benchmarks at all. There's really one thing underneath: a system that changes over time, that you can act on, and that you only ever see part of. Every metric is just one shadow of it — realism checks the frames, rollout error checks the predictions, probing checks the state, and closed-loop checks the planning. Each hides an assumption about when it's even a fair test. Make those explicit, and ask the real question: when does doing well on one metric actually guarantee doing well on another, and when do they just happen to agree?
I'd love to hear what people think, and let me know if you want to collaborate on it.
Understand their concerns but it's not the way…
We’ve identified industrial-scale distillation attacks on our models by DeepSeek, Moonshot AI, and MiniMax.
These labs created over 24,000 fraudulent accounts and generated over 16 million exchanges with Claude, extracting its capabilities to train and improve their own models.
1
260
alireza retweeted
at long last, the final paper of my phd
🧮 Learning to Reason in 13 Parameters 🧮
we develop TinyLoRA, a new ft method. with TinyLoRA + RL, models learn well with dozens or hundreds of params
example: we use only 13 parameters to train 7B Qwen model from 76 to 91% on GSM8K 🤯
alireza retweeted
Simple vision pretraining by predicting next step embedding. The embedding itself is trained along with this while stop grad is applied when it is used as a target.
Some people are unhappy with the AI 2027 title and our AI timelines. Let me quickly clarify:
We’re not confident that:
1. AGI will happen in exactly 2027 (2027 is one of the most likely specific years though!)
2. It will take <1 yr to get from AGI to ASI
3. AGIs will definitely be misaligned
We’re confident that:
1. AGI and ASI will eventually be built and might be built soon
2. ASI will be wildly transformative
3. We’re not ready for AGI and should be taking this whole situation way more seriously
🧵 with more details
1
252
alireza retweeted
Excited to announce our MIT Press book “Neuroevolution: Harnessing Creativity in AI Agent Design” by Sebastian Risi (@risi1979), Yujin Tang (@yujin_tang), Risto Miikkulainen, and myself.
We explore decades of work on evolving intelligent agents and shows how neuroevolution can drive creativity in deep learning, RL, LLMs and AI Agents!
📖 Free open-access edition: neuroevolutionbook.com
In addition to our own works, this video features work by Jürgen Schmidhuber (@SchmidhuberAI), Seth Bling (@SethBling), Igor Karpov, Jacob Schrum, Yulu Gan (@yule_gan), Ken Stanley (@kenneth0stanley), Joel Lehman (@joelbot3000), Jeff Clune (@jeffclune), Nick Cheney (@CheneyLab), Richard Song (@XingyouSong), Chelsea Finn (@chelseabfinn), Julian Togelius (@togelius), Sam Earle (@Smearle_RH), Hod Lipson (@hodlipson), and Jean-Baptiste Mouret (@jb_mouret).
alireza retweeted
✨ New course materials: Interpretability of LLMs✨
This semester I'm teaching an active-learning grad course at @TelAvivUni on LLM interpretability, co-developed with my student @dhgottesman.
We're releasing the materials as we go, so they can serve as a resource for anyone curious about how LLMs work from the inside:
github.com/mega002/llm-inter…
alireza retweeted
What if next-token prediction wasn't a single forward pass, but a tiny optimization problem?
Introducing: nanoEBM a tiny transformer that learns to think harder by doing gradient descent on its own predictions.
You can start training on your Mac now - it comes < 400 lines
alireza retweeted
Tired to go back to the original papers again and again? Our monograph: a systematic and fundamental recipe you can rely on!
📘 We’re excited to release 《The Principles of Diffusion Models》— with @DrYangSong, @gimdong58085414, @mittu1204, and @StefanoErmon.
It traces the core ideas that shaped diffusion modeling and explains how today’s models work, why they work, and where they’re heading.
🧵You’ll find the link and a few highlights in the thread.
We’d love to hear your thoughts and join some discussions!
⚡ Stay tuned for our markdown version, where you can drop your comments!
alireza retweeted
Reinforcement Learning (RL) has long been the dominant method for fine-tuning, powering many state-of-the-art LLMs. Methods like PPO and GRPO explore in action space. But can we instead explore directly in parameter space? YES we can. We propose a scalable framework for full-parameter fine-tuning using Evolution Strategies (ES).
By skipping gradients and optimizing directly in parameter space, ES achieves more accurate, efficient, and stable fine-tuning.
Paper: arxiv.org/pdf/2509.24372
Code: github.com/VsonicV/es-fine-t…
alireza retweeted
much more convinced after getting my own results:
LoRA with rank=1 learns (and generalizes) as well as full-tuning while saving 43% vRAM usage! allows me to RL bigger models with limited resources😆
script: github.com/sail-sg/oat/blob/…
LoRA makes fine-tuning more accessible, but it's unclear how it compares to full fine-tuning. We find that the performance often matches closely---more often than you might expect. In our latest Connectionism post, we share our experimental results and recommendations for LoRA.
thinkingmachines.ai/blog/lor…
alireza retweeted
(1/7) New preprint from the Rajan Lab! 🧠🤖 @RyanPaulBadman1 & @SimmonsEdler show–through cog sci, neuro & ethology-how an AI agent with fewer ‘neurons’ than an insect can forage, find safety & dodge predators in a virtual world. Here's what we did
Paper: arxiv.org/pdf/2506.06981
































