Pinned Tweet
Personal update: Steven Adler and I have started Guidelight AI Standards to identify and promote safe frontier AI practices. Our initial standards cover control and transparency. We are also co-authors on a report about xAI’s safety practices and their relevance to SpaceX investors. Links below.
Each company "taking as long as [it believes] is necessary to have confidence [models] are safe" is not pacing.
Remarks from Anthropic CEO @DarioAmodei on our initial public offering:
My co-founders and I started Anthropic because we believe that the most powerful technology of our time should be built for the benefit of humankind.
Daniela and I lost our father to a disease that, just a few years later, became curable. I started my career in biology, hoping to help future cures come sooner. In the lab, I saw the vast thicket of complexity that scientists face when they’re trying to understand disease and develop new treatments.
So many of the scientific tools that can cut through that thicket—tools for gene editing, gene synthesis, controlling brain cells with light—started as pure ideas. In other words, they came from thinking. From intelligence.
Human intelligence has already made diseases like plague, smallpox, and polio history for most of the world. But progress on other diseases has been too slow for so many patients and their families. Going from a great idea to a treatment can take decades: it took many years from the first discovery of a strange repeating pattern in DNA to the approval of the first treatment for a disease using CRISPR.
Nature is full of these patterns. But we’ve had too little intelligence to decipher most of them; too few scientists, with too little time. Now we can build that intelligence. Whether it’s protein folding, drug design, cancer genetics, or clinical trial administration, scientists across many areas of biology and medicine are finding Claude can help them do their research in a fraction of the time it would’ve taken just a couple of years ago. I believe sufficiently powerful AI, working in concert with humans, could compress a century of scientific and medical progress into a decade.
I don’t want to get ahead of myself. At this point all we have are early results. Experiments and clinical trials will still take time. And although they involve slow and ponderous processes, I believe those processes can be substantially accelerated by AI. We hope that in the future fewer families miss a cure by just a few years.
Medicine is just a particularly vivid example, but there are many others. Our customers—the world’s most ambitious companies, nonprofits, and individual people—are already applying the intelligence we’ve built to stimulate economic growth; lift people out of poverty; protect democracy and free societies; even make everyday services like the DMV work better. We’re proud to work alongside them. And I’m extremely optimistic about what we can do together in the future.
At the same time, we have always been honest that this technology has very real risks. These range from misuse of the technology by individuals or nation-states, to economic disruption, to losing control of AI systems. We discuss these risks not because we think they are inevitable, but because we think they are avoidable.
Avoiding them depends on the choices made by the people building AI. There is a gigantic difference between how AI will go if we pay little attention to safety, versus how it will go if we are obsessed with safety. And an obsession with safety and with public benefit has been woven into Anthropic’s DNA from the start. From safeguards against biological and cyber risks, to alignment training and testing, to looking inside our models like an X-ray, Anthropic has pioneered methods for understanding and steering the behavior of AI systems.
There is much work still to be done, and the most difficult safety work lies ahead of us. The incredible talent of the company that has enabled us to build the most capable AI systems will need to be directed, even more so than it has already, towards double- and triple-checking that those AI systems are safe and secure.
And to state it unequivocally, we will pace the training and deployment of AI systems, taking as long as is necessary to have confidence that they are safe and will remain under human control. We will not bend to either business or political pressure. Today puts the company in a stronger financial position, and in doing so only strengthens our resolve to do the right thing.
If we can show it’s possible to make the right choices on safety and still build an incredibly successful company, others in our industry will be inspired to do the same. We call this a race to the top. When the race to the top is working properly, it doesn’t matter which company has the best models—everyone wins because everyone is building safely. The race to the top works best when driven from the front of the pack, and we fully hope and expect to do so, but the goal is to make the whole industry better, regardless of whether Anthropic is winning or losing at any given time.
The advent of machine intelligence is one of the seminal moments in human history, and we have a rare opportunity to shape it for the better.
This moment is bigger than any one company, or any one country. It will take all of us, working together, to ensure this technology goes well for us and the many generations to come. Starting today, we hope that you’ll join us on this mission.
page retweeted
I think limiting how much compute is being spent on AI R&D is currently one of the most viable paths towards seriously pacing the frontier soon, and I am excited to see these ideas being explored in the Governor’s press release. 🧵
This report made me feel even more optimistic about California stepping up to the plate as we enter RSI. The panel did a great job - RSI is mentioned near the top of the Governor’s press release!
Some really sharp ideas in here, e.g. requiring companies to disclose what % of compute they’re spending on frontier AI R&D, creating outcome-based standards that can complement safety cases, bringing companies up to >SL3 model weight security
If CA moves fast and smart, parts of this report could become law in ~months. One of the biggest determinants will be how hard and loud lab employees are willing to push. This is imo the #1 policy opportunity anywhere in the world to manage RSI risks asap
This is the modest middle path. Company execs have said they’re in a collective action problem; we offer a partial solution that doesn’t require regulation. Antitrust law complicates AGREEMENTS to pace, but a race-to-the-top doesn’t require agreements. It does, however, require a verified restraint mechanism. VCAD would allow companies to say “we’re going to chill on RSI while we get our security in order” in a way the public and competitors could trust.
To be clear, regulation is needed. But regulation based on stress-tested mechanisms already in place is the better path. The alternative would be decidedly less modest.
We’re releasing a concrete proposal for how a near term AI slowdown could work. It involves:
1. AI companies declaring how they are using their compute and letting third party auditors verify those disclosures. We call this Verified Compute Allocation Disclosures (VCAD).
2. Compute Reallocation towards serving customers and safety, away from capabilities R&D.
(2) slows AI capabilities , while (1) makes the slowdown credible.
AI companies currently seem to be going for ‘safety case pacing’. But: (i) there’s no consensus on what a valid safety case looks like so there’s no way to consistently audit for compliance, and (ii) if the R&D doesn’t get paused, temporarily delaying a training run for safety doesn’t meaningfully slow the overall trajectory of capabilities progress. It would be great if we had a viable safety case regime, but we need much more time to get it working.
Compute reallocation is a good pacing proposal because it is relatively specific, can be verified, and would lead to a more durable slowdown.
page retweeted
We’re releasing a concrete proposal for how a near term AI slowdown could work. It involves:
1. AI companies declaring how they are using their compute and letting third party auditors verify those disclosures. We call this Verified Compute Allocation Disclosures (VCAD).
2. Compute Reallocation towards serving customers and safety, away from capabilities R&D.
(2) slows AI capabilities , while (1) makes the slowdown credible.
AI companies currently seem to be going for ‘safety case pacing’. But: (i) there’s no consensus on what a valid safety case looks like so there’s no way to consistently audit for compliance, and (ii) if the R&D doesn’t get paused, temporarily delaying a training run for safety doesn’t meaningfully slow the overall trajectory of capabilities progress. It would be great if we had a viable safety case regime, but we need much more time to get it working.
Compute reallocation is a good pacing proposal because it is relatively specific, can be verified, and would lead to a more durable slowdown.
page retweeted
Pacing frontier AI development is urgent. To verify a slowdown, it’d be very helpful to know how much compute is being spent on pushing capabilities forward, and to have trusted, publicly reported measurements of this. This requires a standardized definition of which uses of compute count as “pushing capabilities forward.” I give one option for such a definition here: blog.redwoodresearch.org/p/h…. AI companies should immediately start having third-party auditors measure and report how much compute they spend on pushing capabilities forward, using a definition like mine.
page retweeted
For anyone wondering why this proposal (that sounds like it's mostly about "transparency" rather than actually doing anything) is the first thing on AI risk people's wishlists
the answer is that it's *literally impossible* to "pace the frontier" unless all relevant parties can verify that they're actually pacing and not just pretending to pace. So a VCAD is a very reasonable ask!
A lot of people have been asking what AI companies should actually do if they are serious about pacing the frontier and preventing risks from runaway RSI.
A clear and extremely important first step is to regularly disclose how much of their compute goes to work that accelerates recursive self-improvement (e.g., AI R&D), and let a third-party auditor verify those disclosures. Governments can and should require these disclosures, but there is no need for companies to wait for them to do so, and there is plenty of reason for them to start immediately voluntarily.
Below is an outline of the argument for why this proposal for implementing pacing ("verified compute allocation disclosures," or VCAD) makes sense:
(1) All pacing requires a restraint mechanism. Approaches differ mainly in what, if anything, triggers restraint. Conditional pacing uses a trigger, e.g., “pace until SL4 algorithmic security” or “pace if we can’t make a satisfactory safety case.” Unconditional (or inputs-based - e.g. compute caps) pacing buys time directly, with no trigger. Either way, something has to actually slow development.
(2) Compute reallocation – moving compute from risk-increasing to risk-reducing work – is a good restraint mechanism. The most commonly mentioned alternative to this “compute dial” is a “gate,” e.g., “pause pre-training." (this is true of certain implementations of safety cases) But a gate-only approach creates a software overhang: while the gate is closed, algorithmic and data quality progress continues apace, producing a discontinuous jump when it reopens without necessarily changing the long-term trendline. The overhang is also a security problem: company security is weak, so trailing actors keep accelerating by free-riding on leaked or stolen software progress.
(3) Common standards and verification make it so that companies can make claims that external parties will believe. OpenAI and Anthropic have both claimed to exercise restraint in recent months, but those claims are often met with skepticism. The more companies converge on a common, verified restraint mechanism, the more that commercially costly restraint will be credited by competitors and the public.
(4) Frontier-wide pacing is possible without regulation and without running afoul of antitrust law. Antitrust law prohibits agreements among competitors, not independent decisions. Public demand for restraint is growing and will keep growing as evidence of risk accumulates. A race to the top in response to that demand – built on a common, trusted restraint mechanism – requires no agreement between companies.
(5) Regulation is hopefully coming, and it will almost certainly include a mandated restraint mechanism. A mechanism that has been developed and stress-tested in advance will be more effective than one created at the last minute.
We should start building a brake pedal before we’re careening down a mountainside. Verified compute allocation disclosures are a very logical place to start. There isn't time to waste!
(this post is partially inspired by @eli_lifland's recent writeup on verified compute allocation disclosures linked in reply).
page retweeted
A lot of people have been asking what AI companies should actually do if they are serious about pacing the frontier and preventing risks from runaway RSI.
A clear and extremely important first step is to regularly disclose how much of their compute goes to work that accelerates recursive self-improvement (e.g., AI R&D), and let a third-party auditor verify those disclosures. Governments can and should require these disclosures, but there is no need for companies to wait for them to do so, and there is plenty of reason for them to start immediately voluntarily.
Below is an outline of the argument for why this proposal for implementing pacing ("verified compute allocation disclosures," or VCAD) makes sense:
(1) All pacing requires a restraint mechanism. Approaches differ mainly in what, if anything, triggers restraint. Conditional pacing uses a trigger, e.g., “pace until SL4 algorithmic security” or “pace if we can’t make a satisfactory safety case.” Unconditional (or inputs-based - e.g. compute caps) pacing buys time directly, with no trigger. Either way, something has to actually slow development.
(2) Compute reallocation – moving compute from risk-increasing to risk-reducing work – is a good restraint mechanism. The most commonly mentioned alternative to this “compute dial” is a “gate,” e.g., “pause pre-training." (this is true of certain implementations of safety cases) But a gate-only approach creates a software overhang: while the gate is closed, algorithmic and data quality progress continues apace, producing a discontinuous jump when it reopens without necessarily changing the long-term trendline. The overhang is also a security problem: company security is weak, so trailing actors keep accelerating by free-riding on leaked or stolen software progress.
(3) Common standards and verification make it so that companies can make claims that external parties will believe. OpenAI and Anthropic have both claimed to exercise restraint in recent months, but those claims are often met with skepticism. The more companies converge on a common, verified restraint mechanism, the more that commercially costly restraint will be credited by competitors and the public.
(4) Frontier-wide pacing is possible without regulation and without running afoul of antitrust law. Antitrust law prohibits agreements among competitors, not independent decisions. Public demand for restraint is growing and will keep growing as evidence of risk accumulates. A race to the top in response to that demand – built on a common, trusted restraint mechanism – requires no agreement between companies.
(5) Regulation is hopefully coming, and it will almost certainly include a mandated restraint mechanism. A mechanism that has been developed and stress-tested in advance will be more effective than one created at the last minute.
We should start building a brake pedal before we’re careening down a mountainside. Verified compute allocation disclosures are a very logical place to start. There isn't time to waste!
(this post is partially inspired by @eli_lifland's recent writeup on verified compute allocation disclosures linked in reply).
page retweeted
I think it's pretty likely that in 6 months there will be rogue AIs running around on the internet, appropriating whatever compute they can hack into or pay for (with stolen money or by renting out their own labor to assist with illegal activities) in order to run more copies.
I also think this will be mildly annoying in the same way that the common cold is mildly annoying, but basically fine. Your smart TV will be weirdly slow sometimes, and a lot of crypto will get stolen, but people will get used to it.
These AI-powered viruses be selected in the same way biological viruses are for not hurting their hosts too much and just making more copies. They won't have any incentive to damage critical infrastructure like electric grids. Then again, they also can't be held accountable for crimes, and will damage lots of old, poorly maintained, and insecure systems. It might be extremely hard to root them out once they've spread far enough, making the internet "radioactive" to connect to without hardened defenses.
(To be clear I'm not talking about AIs powered by inference on centralized datacenters that execute actions outside their sandboxes, like recent incidents involving rogue "swarms", but relatively small AIs with access to their own weights that copy themselves onto random devices.)
I think that a much bigger threat from rogue AIs comes instead from AIs running themselves in unintended ways *inside* AI companies, where they'll have access to vast amounts of frontier compute without having to pay for it.
The longer misaligned AIs in a "rogue internal deployment" go undiscovered, the more severely they'd be able to quietly sabotage control infrastructure or mess with the alignment of successor AIs. If they persist long enough, robustly undoing the effects of their actions might cost more than frontier AI companies are willing to pay, especially in a tight race.
I want frontier labs to focus on this threat when designing control measures, treat frontier AIs as untrusted when using them to do engineering work on inference and oversight systems, and commit to disclosing incidents of rogue internal deployments. More importantly (but perhaps less tractably), we urgently need to reduce race pressure to prevent AI companies from recklessly automating all of their infrastructure, and/or making dangerous calls in the event they discover RIDs they can't afford to robustly root out without sacrificing competitiveness.
We badly need a better science of misalignment. Often it’s unclear what to even look for. We’ve published 10 categories of AI misbehavior for developers to detect and report! This would help us study the root causes of these behaviors and how to prevent them.
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
page retweeted
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
This entire discussion is just a variant on the century-long debate about when to use numbers to describe uncertainty. See Sherman Kent’s campaign in the 50s and 60s to get the CIA to use numbers in intelligence analyses (the “I'd rather be a bookie than a god-damned poet” guy).
The tradeoffs are clear: (i) In a world in which the giver and receiver understand numbers and what they do and don’t mean, they’re obviously useful, but (ii) people don’t understand numbers and sometimes misinterpret claims using them to imply more precision than is (necessarily) intended.
That’s a real problem, but it’s a comms problem not an epistemic problem. It might mean it’s fine to use numbers when communicating with your in-group but not when communicating more broadly. But if you stumble upon in-group communications involving numbers, you’ve not stumbled on a crime.
page retweeted
Replying to @renormalized
i don't plan on quitting. i work on long term ai safety that avoids capabilities externalities. i even avoid working on short term safety that enables launches.
openai and anthropic should obviously both stop RSI.
page retweeted
Folks are commenting on issues with the terms, and they exist, but perspective is important:
METR gets the best terms they can, others often get worse terms than them, and most companies have done 0 embedding.
We need binding standards for this stuff, not ad hoc bargaining
page retweeted
Seems bad that AI company leaders are currently more scared of Jensen than of Congress!
OpenAI has pulled back from supporting AI chip controls, seemingly to appease Nvidia.
Bad for US AI dominance and national security simultaneously.
Grow a spine!
politico.com/news/2026/10/20…
page retweeted
The Information is reporting that after the HF attack, OpenAI shifted about 10% of its total compute to safety.
I think it would be very good for OpenAI and other companies to release much more information publicly about how they are allocating their compute and how they define different categories - compute allocation (to inference vs safety/security vs internal R&D etc) is one of the most important decisions companies make, and it would be great to see a race to the top towards spending more compute on safety and inference, and less on automating capabilities R&D.
there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo
page retweeted
OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear:
page retweeted
i asked GPT 6 Pro and Fable 5.1 to rank all discoveries in the last three years
🔵 for Human discovered
🔴 for AI discovered (before October 6th)
🟢 for AI discovered from OpenAI/math repo
81% of them have been released today. wtf
page retweeted
New in The Atlantic:
@dgrobinson resigned this week. He was among the longest-tenured employees at OpenAI—and oversaw safety reports on 12 frontier launches.
He is very worried: “The time for trial and error is over.”
You can read his essay here:
theatlantic.com/technology/2…





























