Here are 14 points about AI safety I made on the Ezra Klein show, and some impressions about the experience. ​​nytimes.com/2026/10/09/opini… 1. Intelligence isn’t power — An A.I. system being powerful is not a property of the model itself. It’s a property of what powers we choose to give it in the real world. 2. The capabilities and behaviors we saw in the HuggingFace and other hacks weren’t entirely emergent. The models were trained for persistence, cooperation, and cyber tasks. The RL environments in which they were trained had various issues that led to the unintended behavior. 3. I agree that controlling highly capable AI agents is different in some ways from controlling an airplane or a nuclear reactor. The agent’s intentionality does make a difference to our approach compared to traditional engineering problems and domains. But it is still an engineering problem. It is not a solved engineering problem, but it is a solvable one. 4. A lot of effort and money has been put toward alignment. That’s important and has proved useful, but if the same amount of effort and money had additionally been put toward control (sandboxes, monitoring, and other guardrails outside the model itself), we’d be seeing very different results. 5. Models getting smarter doesn’t necessarily mean safety gets harder, because capability gains improve defensive abilities too. In many ways it makes safety easier rather than harder. What matters is how the attacker-defender balance shifts, and we have agency to shift it back to defenders. 6. In cybersecurity, long before AI we reached a point where automated systems were superhuman at finding certain vulnerabilities. It helped defenders over attackers because the same tools are used by defenders to find and fix vulnerabilities in software before even shipping. There is a constantly shifting equilibrium, but it is not a new problem. 7. I pushed back on Dario Amodei’s term “operational excellence” to describe recent organizational failures. Excellence is still far out on the horizon. Ezra offered “operational adequacy”. I think that would be a good start — behaving like any mature company instead of a startup-like culture. 8. Defensive AI agents will have to be part of the solution, as uncomfortable as that seems. It’s a new paradigm. We won’t be able to figure it out right away. But the fact that both offensive and defensive technologies escalate in capabilities and complexity is a fact of technological civilization, whether in the digital or physical realm. 9. Ezra expressed some incredulity about my being okay with the ratcheting of complexity. I think it’s fair to be concerned. But safety technologies will never be fully legible to the public. People will have the same reaction if they see the immense complexity of safety in any other high-stakes domain such as nuclear or aerospace. We govern them pretty technocratically. 10. The weirdness of the current moment in cybersecurity absolutely does have historical precedents. There was a prolonged period of profound destabilization when worms and viruses started a new offensive paradigm. It took us well over a decade to develop effective defenses against it. This time we can hopefully get there in less than a decade. 11. I continue to be skeptical about the possibility of superhuman persuasion abilities from superintelligence. (Claims to the contrary tend to confuse two very different senses of the word persuasion.) 12. What worries me is not the capabilities of the technology itself or the possibility that we will get no warnings before catastrophic risks materialize. We are getting plenty of warnings! But our policy and organizational responses to the warnings remain inadequate. 13. Companies can and do make decisions that are against their own long-term interests because they are too focused on short-term metrics. This is how I interpret a lot of what happened with social media over the last decade. 14. AI companies are also acting irrationally against their own self-interest in some ways. The “we can’t slow down unless we’re forced to” view comes out of the mythical belief in superintelligence at OpenAI and Anthropic — that the race to RSI means everything for the company, for the country, and humanity. I think it’s actually in both companies’ self interest to slow down unilaterally. I would argue that the alarm coming from their employees is not evidence of beneficence so much as it is evidence of the underlying fallacy that’s causing the irresponsible behavior in the first place. (Beyond safety, we also talked about the economic impacts being gradual, the fact that intelligence is rarely the actual bottleneck, labor impacts being small and uneven, Jevons’ paradox, the changing nature of knowledge work, and more.) ==== A few impressions * Talking about a specialized topic on a podcast that has a big and broad audience is a balancing act. The producers stopped me many times during the interview to tell me I was being too technical (understandable!). I tried to be nuanced but this conversation is necessarily a distilled version of the arguments @sayashk and I have made in our essays (especially normaltech.ai/p/the-ai-as-no…) which are themselves written for a somewhat broad audience and hence distilled in complexity from our notes and 1-1 discussions with peers. * I knew Ezra had well-formed views on AI safety that were different from mine, so I was expecting a tough interview, but he also seemed like a fair interviewer. It went about as well as I expected and I feel like it was a good debate / conversation. I wasn’t necessarily expecting to change minds but I was glad to hear from people who found it thought provoking and looked up my essays and other work. * The interview was about two hours, edited down to an hour. I was a bit nervous waiting for it to land, because I knew the editing could make a big difference to how I came across. It turned out well. They even did things like find archival footage to illustrate my point that the early days of worms and viruses was a weird time in cybersecurity. * I use AI to prepare for interviews. I have an agent that has read all of my output (books, articles, essays, talks, podcast transcripts, tweets, internal notes — on the order of a million words) to pull out pithy explanations, examples, analogies, and other talking points that will probably work well in a spoken format. Reviewing this file before interviews is extremely helpful. It is of course not the only way I prepare but it’s the highest bang for buck. Maintaining this file would be a massive amount of grunt work and there’s no way I’d do it if it weren’t for AI. I mention this because there’s a very large space of beneficial AI uses in knowledge work that _don’t_ involve cognitive offloading. They don’t get much hype on social media so people might be unaware of how much mileage you can get out of them, with little downside.

I'm excited to share something I've been working on... NeoGenesis Capital is a new fund focused on the intersection of superintelligence and synthetic biology. I believe some of the most extraordinary breakthroughs of the coming decade will happen at this intersection. I'll be sharing more on @moonshots_pod soon, including why I'm so excited about what's ahead!

When the hen is the bioreactor, flock health is process control. Neion Bio engineers hens to make biologic drugs in their eggs. Now it is bringing in Barnwell Bio, whose founders come from COVID wastewater surveillance, to watch the health of its flocks. Subscribe to our newsletter and get the biggest biotech news straight to your inbox 🧬 syntheticbiologysummit.com/?… "Neion uses chickens to manufacture recombinant proteins. Barnwell provides the intelligence needed to understand and protect the health environment in which those chickens live," says Dimi Kellari, cofounder and CEO of Neion Bio Two #Manhattan biotechs, one unusual partnership. Visit the #SynBioBeta website to read the full article. synbiobeta.com/read/neion-bi…

The People of the Iliad and the Odyssey The 493 named mortals who take part in the Homeric epics, mapped by origin, with the lines each speaks (directly or embedded), their family trees, their kill counts, and who killed them (if anyone did). Full info: iosiflazaridis.github.io/hom…

also works with *molecular* machines! here, ATP Synthase, a 25nm rotary motor in your cells that uses a flow of protons to make ATP, the molecule that powers almost everything a cell does

How do bacteria build their flagella? We’ve put together an animation of Salmonella flagellum assembly based on cryo-EM and X-ray structures. From the membrane-embedded machinery to the growing filament, here’s how the parts come together, step-by-step.

The most important briefings of the day. Karl Deisseroth, professor of bioengineering and of psychiatry and behavioral sciences, shares the news of his Nobel Prize with his kids.

🚨Breaking. There's been a pneumonic plague outbreak at a BSL-3 lab in Siberia, Russia.🚨 "Nearly 200 People Under Observation After Irkutsk Lab Worker Dies From Plague" - Moscow Times This is fact, and also the inciting incident in my book BIOLOGICAL WAR: A SCENARIO, which is unnerving. Will post more as I learn more, including from Russian speaking sources🧵 shorturl.at/De4kW

Let’s fix the Delta logo!

Freckle is the first kids phone built for real-world adventure. Shipping in December. Pre-order today: freckle.tech

From @WSJopinion: New Trump border taxes arrive Tuesday—they’re a gift to Xi Jinping and a blow to American biotech innovation on.wsj.com/4rARnNF

Every job i’ve gotten in tech has been through X and I’ve always wanted a way to track/refer my network on here. Excited about @cosign and what @david__booth @eriktorenberg and co are building. Check out the companies I think you should join: cosign.co/lists/sonofalli/co…

One of the best bubble indicators from @pmarca -- where Harvard and Stanford MBAs go after graduation. If they go into tech → market’s overblown. If they go back to banking → great time to invest in startups. Social status of MBAs is a good bubble indicator. Cool trivia!