Agreed that criminal liability doesn't protect us from this. Civil liability so victims can be made whole is a valuable governance tool, but to the governor's point: if this happens at scale, the AI company could go bankrupt. This is the importance of making AI companies carry insurance for harms that go past bankrupting them
Imagine a rogue AI agent liquidating your bank account and investment portfolio.
If you are the only victim, I suppose you could get made whole by the AI company responsible.
If rogue agents liquidate hundreds of billions across millions of accounts, the AI company would go bankrupt and you wouldn’t get made whole.
Imposing criminal liability on AI executives might help mitigate it at the front end, but once the crime is complete and you lose everything criminal responsibility will be necessary but completely insufficient.
Please make congressional technical understanding great again
Excited to share this updated primer on the Office of Technology Assessment (OTA) with @danielschuman. With new economic/cyber/bio/and other policy challenges arising from AI, now would be a great time to invest in greater science and tech capacity in Congress.
cc @SpeakerJohnson @hakeemjeffries
"Every facet of the federal government, and even a lot of state governments, will benefit from hiring scientists into policy support roles directly instead of interfacing to them through third parties."
This is a very good article.
I'm in AI Frontiers today arguing that countries and companies need to wake up to AI-driven takeover of outer space.
Due to the compounding returns of being the first mover and tapping into massive infrastructure in space, a winner-take-all outcome is looming.
Rival states, who lag in launch capacity as well as frontier AI, would have an incentive to strike first, before such a pioneer establishes enduring hegemony.
Here's why the first-mover advantage could be so durable:
First movers in space will benefit from an industrial flywheel, wherein space-based solar power enables more compute and the construction of yet more space-based solar and thus more compute and furthermore.
Moreover, many of the engineering problems that make spacecraft fragile and costly today—and which cause people to instinctively dismiss rapid space takeover as science fiction—will be solved using AI-accelerated engineering. Brutal present-day latency in robotic operations (e.g., a radio signal can take 40 minutes to and from Mars...possibly for a single instruction) will be circumvented by on-board intelligent operation (i.e., full self-driving for orbiters, landers, and rovers).
The result is that a lead in AI capabilities could have huge, huge industrial returns for an actor that first manages to spin up infrastructure in space. That includes the orbital analogs to workshops and shipyards.
Laggard countries and companies simply will not benefit from these dependencies, such as orbital factories and depots for assembling structures that wouldn't fit into one rocket fairing (e.g., too wide for the hold of Starship) and which therefore would be infeasible for rival states to produce.
And herein lies the wicken security dilemma:
The optics for laggard nations are awful. As in MAIM, a country just isn't used to thinking about kinetically striking foreign soil merely to stop a commercial computing operation or, in this case, because some really cool and innovative company is collecting energy and putting lasers and robots into space. It feels wholly inappropriate to exert violence on a bustling business.
But such a bustling business is exactly what we will see as one actor (whether private or government) is spinning up history's largest industrial flywheel—extraterrestrial infrastructure providing energy, compute, and robotic manipulation to carry out all its other ambitions in space.
Finally, the holy grail of space expansion will be to launch von Neumann probes or specialized development convoys into space to establish colonies and proliferate themselves. In other words, a space virus.
Hmm...how could this ever go wrong?
1) If these AI envoys are aligned to the interest of just one man, other countries had better hunt that shit down.
2) If these AI envoys are hastily, sloppily developed and thus misaligned with civilization overall, we had better hunt that shit down.
Nation states must wake up, because there may be a point of no return.
Blaine Dillingham retweeted
Exciting to see this model announced with an Apache 2.0 license. It's been over a year since an American developer has released anything this capable and permissive.
Congratulations to the @reflection_ai team on this release!
Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active.
- Frontier reasoning efficiency
- Advances the Western open frontier on coding & agentic tasks
- Trained end-to-end from scratch
Full weights release this month.
Learn more about Beam: reflection.ai/beam
This. I would just note that liability alone can’t handle the tail risks Bessent mentions, as labs might be bankrupt before they can be held responsible for the full harm they caused. Insurance mandate helps address this. It also gets us embedded inspectors who are hired by insurers, rather than just METR or other anthropic friends
If only there was a way to get auditors with good incentives…
ai-frontiers.org/articles/do…
And they said an actuarial model wasn’t something to get excited about
There will be watch parties all around the world on release day
The Artificial Intelligence Underwriting Company has raised $55M to audit and insure frontier AI models.
Insurance produces audits the public can trust and understand. The mechanism is simple: insurers need audits to price the risk. They pick auditors they believe will help them price the risk most accurately. Insurers then turn the audits into a price signal that cuts through political debates. Insurers are aligned with the public: if they get the audits wrong, they go out of business. That’s different from today, where labs pick their own auditors, creating public distrust.
Insurance only works if we scale up auditing, and we are building the infrastructure to do that. Millions of agents are being deployed; they leave terabytes of traces and will create thousands of incidents across dozens of risk categories. To scale auditing, we need reliable agents to surface evidence traces for human review.
History guides us. Take cars: auto insurers funded crash tests to price risks; crash tests led us to airbags, lowering the risk. Take electricity: home insurers funded lightbulb tests to price fire risks; today, every lightbulb carries a safety mark. Nuclear safety works similarly.
We’ve been learning by doing.. In the last 6 months, we’ve audited and insured frontier agent builders like Cursor, ElevenLabs, Lovable and Harvey. For ElevenLabs to get the world’s first agent insurance coverage, a Lloyd’s of London insurer required quarterly technical audits by an auditor of the insurer’s choice. We’ve built infrastructure for quarterly technical audits of any agent, working alongside firms like Gray Swan (technical evaluations) and Schellman (audit).
In the coming weeks, we’ll share our vision for how an insurance ecosystem can work for the frontier, and open source the world’s first actuarial model for frontier AI catastrophic risk.
We’re hiring across all roles in San Francisco. Come do your life’s work with us.
Rune & Rajiv
What if instead of trying to regulate AI the same way we regulate viagra, we recognize the need for a thoughtful governance regime that draws precedent from the nuclear, cyber, aviation, and satellite industries on how to mitigate risks in highly technical domains ai-frontiers.org/articles/do…
Ah yes, an “FDA for AI”. Because we can just run randomized controlled trials on agent swarms
What could go wrong when evaluating Highly Persistent Internal Model for Safety and Efficacy
It seems to me like the ideal model for AI governance would be embedded auditors from the government, perhaps from a combination of CAISI, the national labs, nat sec world, etc. who don't have any direct regulatory or other coercive authority, but who can flag things to the executive for emergency concerns and who generally have good awareness of what is going on. And then perhaps we have some kind of well defined procedure with careful judicial review to get temporary restraining orders or preliminary injunctions for imminent harms, medium term injunctions for fairly pressing harms but where we don't want to burden the companies for a super extended period of time (The FRONTIER Act does a version of this). And then all of this exists alongside the insurance infrastructure, and the insurers can use the data that comes from the national labs or CAISI or the Natsec world as they see fit in the same way that the Nuclear Regulatory Commission does a bunch of in-depth inspections, and a lot of that information is used by nuclear insurers to price risk.
Another helpful inspiration from Nuclear world could be INPO, an industry consortium that also does detailed technical inspections that inform insurers. There are spillover benefits to safety research, which means frontier labs might be underinvesting in it because they can’t internalize the positive externalities. Allowing them to form an industry consortium to pool resources toward alignment and security research can help address this. But we shouldn’t make it a self regulatory organization, or give it any kind of binding authority.
Alongside all of this, I’m open to IVOs that audit for 1) risks that are uninsurable because they’re too large of tail risks and 2) risks that aren’t really torts or are poorly handled by ex post liability (surveillance/leverage/persuasion by the labs, as an example). But this may not be worth it, as I don’t want to later on an insane amount of regulation.
I hear lots of AI policy folks say the problem with an insurance mandate is that we don't have time. “Timelines are too short” and while insurance might be theoretically good, in practice the IVOs might just be good enough. I’ve come around to thinking they might be correct. But I guess what I’d say is: let's not let this obscure the fact that the IVO design is deeply imperfect (race to the bottom), and the insurance design is much better, though still not a cure all. So the upshot is that we should implement whatever policy we can get in the immediate term while continuing to work towards good institutional design. That is, we shouldn't let the stopgap of FINRA or IVOs or whatever become sticky. And the fact that it probably will is a huge risk to be aware of
Very much appreciate someone laying out the stopgap plan that we urgently need. Our governance of AI can have stages: I remain skeptical of IVOs’ incentives, and eventually we’ll need something better-designed like a catastrophe insurance mandate for frontier labs. But IVOs could be helpful in the short term before they get captured or fall to bad incentives. It seems like a reasonable order of policies is:
ASAP:
-enabling the frontier labs to coordinate a slowdown by having the Admin promise narrow protection from antitrust scrutiny
-the phone-call-based governance Anton lays out where the Admin tells labs they need to have METR, Redwood, etc. red team their systems
Short term:
-formalize that requirement for third party audits, probably by EO, whatever
-Finra for AI or whatever. Not a particularly great design I don’t think, but better than nothing.
-continued antitrust shield to coordinate a slowdown if needed
“Medium” term (6mos - 1 year):
-ideally and (very doably), get insurance ecosystem up and running to create the right incentives for third party auditors
-if that can’t happen, IVOs are second best
Medium term +:
-we really should design institutions with better incentives than IVOs. Stopgaps are fine, but this technology could be too consequential to leave it up to “I trust METR to do a good job.” METR is great! But organizations change when you give them new jobs and incentives! Gabe Weil’s catastrophe insurance and liability work is the best idea i’ve seen for incentive design, and we will really need that. If for whatever reason (perhaps political) we can’t do that in the immediate future, that’s fine, I’ll take “Finra for AI” or IVOs or whatever. But we shouldn’t let those models become sticky! Eventually, we need those IVOs or whatever they’re called to be hired by insurance companies that actually have a financial motive to get the audits right. So to anyone who says “timelines are too short for insurance”: you’re probably right. That means doing non-insurance policies asap. But it doesn’t change the fact that ultimately insurance is better-designed than IVOs and we should implement a well-designed system with good incentives as soon as we can. We can do X, Y and Z (non of them insurance) to prevent “doom” in the super short term. But if it turns out we don’t all die in the next year, then we can do the right thing and do insurance!
“Long” term (more than a few years away):
-the world might be super changed, insurance won’t be enough, and we’ll need lots of ideas for different institutional designs
-or maybe progress and disruption from AI will be less than expected. Even here, insurance isnt a cure-all, and we’ll need to do more than that. This scenario just means we have more time to do good stuff, and hopefully can do better than haphazard stopgaps
This quoted post is unavailable.
Hearsay, character evidence, unfair prejudice — many rules of evidence are partially motivated by distrust of jurors to properly weigh the strength of different facts. If we could build hyper-rational AI jurors or adjudicators, much of the court system might be worth reimagining
Blaine Dillingham retweeted
Oops. Should've said ~4 months
Blaine Dillingham retweeted
The reason that METR and Redwood are preferred for the Hugging Face investigation is that this incident is much more relevant for its alignment implications than its cybersecurity implications. Everyone gets hacked by external actors; we don't need crash public incident reports for most of those. Not everyone gets hacked by internally deployed models that also autonomously hack external partners.
Accurate decomposition of agent behaviors and analysis of how OAI's deployment configurations and testing practices may have contributed to reward hacking/emergent misalignment are the key objectives of this investigation -- not description of where OAI's DLP, EDR, and reverse proxy systems were poorly engineered.
But yes, METR, Redwood, and Apollo should partner more closely with/hire from Crowdstrike, Mandiant, Deloitte, etc., to upskill in cybersecurity (and vice versa).
