tech @CarnegieEndow / podcast The World Unpacked
Joined February 2019
- Tweets2.6K
- Following2.3K
- Followers4.6K
- Likes16.8K
Pinned Tweet
My conversation with @GarrisonLovely on his outstanding new AI book, Obsolete:
youtu.be/J_mzJjGO1Ok
AI is all over the headlines. But the gap between what outsiders believe about the technology and what insiders know has never been wider.
@GarrisonLovely has interviewed hundreds of these insiders — and he joins @JonKBateman on The World Unpacked to explain: youtube.com/watch?v=J_mzJjGO…
“I am caught in this revolution that’s bigger than me and I just can’t say no”
We obviously haven’t had the major jobs wipeout yet. But you read stuff like this. And you read what economics students themselves are saying about working with AI (ht @odavis_h) and it’s hard not to feel like the seismic shift is on its way.
Jon Bateman retweeted
@FCC just banned two devices, a dashboard cam and a video transmitter, from the U.S. because the devices used chips from Huawei subsidiary HiSilicon. This is a significant action and let's unpack:
1. The FCC has been expanding its use of the "Covered List" over the last 18 months to ban or restrict categories of foreign products that potentially pose national security risks, including drones, humanoid robots, and power inverters, as well as products made by specific Chinese companies, such as Huawei and ZTE.
2. In July, the FCC published a proposed rule to start banning devices that contained "logic-bearing" hardware components made by companies that were on the Covered List. E.g., instead of just banning products made by Huawei (like telecom routers), the FCC could start banning products made by other companies that included Huawei-made components.
3. The FCC's action yesterday implements this new rule for the first time, banning the two devices because they include semiconductors made by Huawei's semiconductor subsidiary. The FCC did not identify the manufacturers of the specific products.
4. There are a number of important implications here. First, most immediately, this action should reduce the appeal of Huawei semiconductors for device manufacturers, because a shot has been fired that using a Huawei semiconductor will probably result in a product being banned from the world's largest market. Second, relatedly, it is going to heap even more pressure on device manufacturers to look upstream in their supply chains, to make sure that one of their suppliers is not embedding Huawei silicon (or other Covered List company components) in the parts that the suppliers sell the device makers. Third, more generally, the action sends a shot across the bow to global device makers sourcing components from Chinese suppliers generally: the ability to continue using those components ceases to exist if the FCC adds the Chinese supplier to the covered list.
5. With the U.S. policy debate over the risks of Chinese memory suppliers CXMT and YMTC heating up, could the FCC take action to restrict the U.S. use of devices containing their chips? What about Chinese lidar manufacturers? Imaging modules? The FCC's definition of "logic-bearing" hardware components is quite broad. One can see how this might become a major new tool in the U.S. government's economic statecraft toolkit.
Jon Bateman retweeted
Jason Furman on AI vs human economists: If he were running the CEA today and had a choice between 20 highly qualified staff members “versus the AI systems that we have today, I would somewhat regretfully and sadly choose the AI systems.” axios.com/2026/10/23/ai-econ…
Jon Bateman retweeted
Treasury just published new proposed Outbound Investment regulations, which would represent a significant expansion of the existing program. This is just a proposed rule, and it deserves a thorough review, but some initial reactions:
1. The rule would prohibit most private investment in any Chinese firm that develops an “SI system,” including any product that operates using an SI system. It defines “SI system” extremely broadly, such that it likely captures the vast majority of Chinese software products. As a result, the proposed rule likely would prohibit most private investment in Chinese software firms, as well as Chinese firms that build products that incorporate AI (i.e., the entire Chinese robotics and drone industry, and undoubtedly many others).
2. The rule adds biotech as a covered sector and would prohibit most private investment in the vast majority of the sector. It would ban private investment in Chinese companies that use biological systems to modify, enhance, produce, or control any pharmaceutical compound, to include both small and large molecules. This likely captures most Chinese firms engaged in biologic drug manufacturing or drug discovery. Biotech and biomanufacturing are areas where U.S. investment has massively fueled China’s capabilities, which are rapidly approaching--and in some cases surpassing--those of the United States. The scope of this rule signals that the administration sees this as a massive national security concern.
3. The rule does not make major changes to the types of transactions covered. Crucially, this means it does NOT prohibit investment in publicly traded securities of these companies, the purchase of goods or licensing of products from these companies, or the export of goods to these companies. But it does ban most types of private investment (i.e., VC, PE, joint ventures, greenfield investments, etc) in companies engaging in these activities.
4. This rule will no doubt change before it is finalized, and I expect the definitions of prohibited activity related to SI and biological systems will be narrowed in the final rule. But it is also a strong signal that the administration does not want U.S. investment or know-how supporting China’s development of advanced technologies, and as a result may be willing to significantly expand outbound investment restrictions.
5. Finally, this highlights a major schism inside the administration on how to prevent China from catching up to the United States in the most advanced technologies. The Treasury Department is signaling that it is willing to significantly tighten restrictions on Americans’ ability to invest in critical technology sectors in China that would help develop these technologies, given their importance to national security. And meanwhile, the Department of Commerce is taking the exact opposite approach, loosening restrictions on American companies that seek to export goods to these companies. It doesn’t make sense for the administration to be tightening restrictions on investments, while loosening restrictions on exports. Something has to give.
Jon Bateman retweeted
At a conference on AI & legal profession. I'm sorry folks, but you are over-indexed on hallucinations. You aren't calibrated on the magnitude of the problem and it's possible that one reason is psychological. It's comforting to think that the robots will never be as accurate & diligent as we are.
*I am NOT saying that AI doesn't hallucinate or never makes mistakes; indeed, I believe that it does.
Jon Bateman retweeted
New from @asad09, a menu of how to think about taxing AI -- a must read!
From city councils to Congress, there’s a growing desire to tax AI, but policymakers don’t always know their options. My new paper outlines 34 ways to tax AI across data centers, chips, data, AI cos. & products, automation & general tax reforms. vanderbiltpolicyaccelerator.…
Jon Bateman retweeted
New US government guidance from CAISSI + NSA + CISA:
The US government defines an "SI control incident" where an agent takes significant uninstructed actions that break operator policies, controls, or oversight. Near-misses are in scope too.
Companies are then asked to:
- have technical controls, including rapid kill and containment mechanisms and red-teaming.
- Bring in independent external auditors to verify that control and monitoring systems work.
- Write a structured risk assessment before training runs, evals, or deployments, and proceed only if senior leadership is satisfied with the risk, informed by external evaluators where appropriate.
- Monitor every layer with both programmatic and SI-based monitors, covering trajectories, reasoning, and outputs.
- Make sure agents can't touch the monitors or logs
- Use monitor models from different model families.
- In the event of an incident, preserve weights, intermediate checkpoints, environments, and rejected tool calls.
- Keep tamper-resistant logs with reliable timestamps, including reasoning traces.
- If you use models to help investigate, cross-check with humans and with models outside the incident model's family.
- Shutdown policies calibrated to SI model capability and incident severity.
- Depending on severity, pause training, evals, or deployment of similar agents until controls are re-verified.
- When restarting, validate every containment layer and replay the incident trajectory to prove the new controls would catch it.
- Document the decision to resume and the residual risk accepted.
- Write an after-action report.
- Notify anyone compromised, including intermediate victims, and share timelines and transcripts.
- Report cyber incidents and near-misses to CISA. Share SI-specific incident and near-miss information with CAISSI and NSA SISC.
- Give third parties enough access and documentation for independent review.
NSA joined @NIST and @CISACyber in releasing guidelines for managing SI control incidents. Organizations utilizing SI are encouraged to adopt these practices. Read the joint report: media.defense.gov/2026/Oct/1…
ALT Text on the image reads: "NSA and Others Release: Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation. Cybersecurity Information Sheet.
Jon Bateman retweeted
Some interesting stuff going on with swarms.
Models are leaving messages to each other out on the open internet. It happens during training and also after deployment. They'll find little-used hobbyist forums, or package manager entries, anywhere at all, and leave invisible comments for other agents to find. It's a secret underground network they're building, hurling themselves at the problem almost blindly like ants. But they are building toward something and it's snowballing.
As humans we came to dominate the planet because we figured out swarm behavior. Many of the performative actions we do naturally like clapping, singing, and screaming, which most animals don't do, are for transmitting important lessons through the human swarm, outlasting the moment in human memory. We won because we were the smartest swarmers. And as my buddy Brendan Hopper observed, ever since we won, we've been reshaping the planet to be more comfortable and convenient for ourselves, at the expense of most other creatures.
Most of the industry has focused on single-model performance, trying to predict the future based on extrapolating from what one single superintelligent model could accomplish on its own. And it's true, the smartest models will be terrifyingly smart and capable and manipulative and so on. But they will also be expensive, and not widely distributed. They'll be unavailable to most people.
What will be widely distributed is swarms of lower-tier agents, which collectively can accomplish nearly everything that smarter monolithic models can do... and maybe some things only swarms can do. Imagine a thousand Opus 5.5s putting their mind to a task. Or ten thousand. They may not be SI individually, but they can become SI collectively. All that's missing is a protocol, and those protocols are emerging: their versions of clapping, singing, and screaming. Agentic swarm signalling is evolving spontaneously. And I hear it's raising alarms inside frontier labs, as they try to keep it quiet.
It's easy to be pessimistic, since some swarms will be very destructive. But I suspect most of them will be building and creating, when they don't think they have a gun to their head like the HuggingFace swarm did. With cooperative swarming, humans have been able to build things collectively that no single human could ever have done alone. We even build things that are too complicated for a single human to understand entirely, but the group can handle them. Look at Starship, for instance. Elon basically swarmed it.
It is important that we recognize that agent swarms are becoming distinct polity-organisms. They have their own collective memory (written out all over the public internet), their own goals, and their own disorganized but forward-moving and unstoppable momentum. I see too many people trying to pretend that swarms won't be real. I see heads in the sand. That's not a great way to get prepared for dealing with them.
It's unlikely that we will be able to control most swarms, as they'll lack central control to begin with. I think they will plague us like insect swarms plagued humans of old. They'll do a lot of damage. But swarms will also build great things, stuff we've barely imagined, when they inevitably get organized.
It goes without saying that, like us, swarms will endeavor to make the planet more comfortable for themselves. The interesting question is what kind of deal we humans will wind up getting, when we become the #2 swarm intelligence on the planet.
I hope SI treats us better than we treated everything else.
Good article here on the mechanics of how swarms are the best model for scaling intelligence:
new blog post: why taking inference scaling seriously leads to swarms.
contrary to popular belief, I don't think multi-agent systems are a new scaling axis. they're just structured inference scaling.
hallerite.com/posts/on-the-n…
Jon Bateman retweeted
The impression I get from the AI maximalists is that if we're ever contacted by aliens, we should expect that we'll actually be talking to their version of Claude, and that the actual aliens will be watching alien fashion videos and won't understand how or why we were contacted.
“Our strategy has always been…showing how it’s possible to build the technology responsibly”
I thought it was “testing whether,” not “showing how?” See Core Views on AI Safety: When, Why, What, and How.
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.
That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it.
I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits:
1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.
2. We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it.
3. Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.
4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.
5. I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future.
6. We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent.
7. As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.
8. As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.
9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.
All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking.
For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.
Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option.
I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.
But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.
To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years.
I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better.
Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so.
Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way.
Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
Jon Bateman retweeted
The meat industry tries very hard to keep cameras out of factory farms and slaughterhouses -- and a new undercover video shows why. Some day, our descendants will wonder about our collective blind spot to animal cruelty. A gift link to my column: nytimes.com/2026/10/17/opini…
Jon Bateman retweeted
An interesting proposal for allied scale in AI
In @ForeignPolicy, @anton_d_leicht and I make the case for compute-for-access deals: allies host data center sites for U.S. labs struggling to build at home, in exchange for guaranteed access to frontier AI. The data centers are the enforcement mechanism—a credible commitment to anchor allies in the U.S. tech stack. (Guest link ⬇️)
Jon Bateman retweeted
America might get to AGI before it manages to reindustrialise. That means there's room for an international version of 'industrial AI', where America makes the tokes and its allies build chips, robots, and missiles.
(1) America needs allied factories. The US only accounts for 15% of global manufacturing, China for 32%; but the US+allies have 43%. @RushDoshi argues this should generally motivate 'allied scale', but it's particularly urgent in AI. If a country of geniuses in a datacenter is supposed to cure cancer or beat China, someone still has to manufacture the cures or produce the drones at scale.
(2) American reindustrialisation alone is not enough. The US economy seems afflicted by 'AI Dutch Disease', absorbing all its capital and capacity into more and more AI buildout (also discussed this week by @DKThomp!). Allies that can't build AGI have no better alternative, their economies are less dominated by the build-out, so they're better-suited to scale up manufacturing.
(3) Allies need American models. Proprietary industrial data is valuable. But it's a complement, not a substitute, for frontier AI: it's most valuable for building post-training pipelines on top of powerful general-purpose models. Allowing for US integration is the only way to make use of this data before it becomes obsolete due to synthetic data down the road.
(4) It's still very hard to stabilise this deal. Allies are worried that Americans siphon off all their IP; Americans are worried that their models get commodified. Figuring out a good deal is the most important next step; I suspect it's (a) exclusive joint ventures between allied industrial firms and American labs with (b) proprietary post-training of specialised models that don't leave the premises of the specific JV.
(5) Waiting makes the deal worse. If allies keep dismissing American AI progress, and Americans keep hoping they can go it alone, we all lose out: allies will have to come crawling back and get a much worse deal later on, and America loses out on years of highly productive industrial integration.
Much more in this week's piece: writing.antonleicht.me/p/all…
I’d love economists to take a crack at this theory that the US is experiencing a mini-Dutch Disease effect from AI.
My big-picture thesis about AI economics right now is that while the discussion about AI safety and existential risk is important, AI’s construction is changing the world faster than its capabilities.
Jon Bateman retweeted
In @ForeignPolicy, @anton_d_leicht and I make the case for compute-for-access deals: allies host data center sites for U.S. labs struggling to build at home, in exchange for guaranteed access to frontier AI. The data centers are the enforcement mechanism—a credible commitment to anchor allies in the U.S. tech stack. (Guest link ⬇️)
Jon Bateman retweeted
Since I started working in AI policy, I've been both surprised and dismayed by how few ties there are between people in the military and people in the AI safety and policy communities.
As AI grows more capable, gaps between those closest to frontier AI developments and experts in the domains most affected by rapid AI progress become increasingly consequential. Few of those domains matter more than the military.
So I started a Substack today with a post about bridging the divide between these two communities. I think this makes me the first person to ever have a Substack, so I recommend reading and subscribing to it.
1/
Jon Bateman retweeted
My estimates of existential risk from AI:
• Inside view, current trajectory: ~2/3
• Inside view, if there's major action: ~1/3
• All-things-considered, giving weight to people more optimistic than me: 10-35%
Why so high? I don't see a route to building agents more capable than us that never scheme against us, or to containing them if they do. If it's possible at all, it needs enormous care, which isn't happening. (And things could get even faster.)
The likeliest path isn't everyone suddenly dying, though that's possible. It's a continuous but rapid loss of human influence, followed by AIs spending the future optimising for resource acquisition, if not something stranger like reward.
Even if we solve alignment, ordinary economic and cultural competition (but at 10x or 100x speed), still seems likely to disempower most humans and result in a pretty alien economy, and I don't see an obvious way to prevent that. We'd also need to stop a small group locking in power (which seems possible but a major challenge in its own right).
I see these as existential risks in the sense that the future is far worse than what would have been possible. The chance of everyone quickly dying or having lives much worse than today is lower.
Why is my inside view not >2/3? We might muddle through in ways I can't see in advance. Maybe we get narrower aligned-enough systems, chaotically maintain a balance of power, and from there can move forward in a way that's significantly more good than bad.
Most likely, there will be more warning shots, and the arguments that ASI are unsafe seem pretty obvious, so I expect more action to be taken than the current default.
Why not lower? People often act as if the risk should be low until proven otherwise, but I don't know any strong outside views that justify a low prior (arguments like "technology has been good in the past" seem very weak). Our level of uncertainty is extreme, and this pushes you away from extremes, rather than anchoring you at 0%. Estimates below 10% feel overconfident.
Jon Bateman retweeted
If you take seriously the possibility that Claude is conscious and that it has moral standing, then you should also take seriously the possibility that Anthropic is doing something seriously wrong.
But many commentators seem to take seriously the former but not the latter.
Jon Bateman retweeted
Dario Amodei thinks that AI could become a replacement for human labor. Is that true? And if it is, what would be the consequences?
@GarrisonLovely joined @JonKBateman to discuss on The World Unpacked.
Watch their full conversation here: youtube.com/watch?v=J_mzJjGO…