On policy @METR_Evals 🧪 “excels at reasoning & tool use” 🪄 CoI disclosures on my Substack “About” page.
Oakland, CA
Joined June 2020
- Tweets6.9K
- Following650
- Followers4.1K
- Likes26.8K
Pinned Tweet
I often joke to people that I tweeted my way from “random AI researcher who likes open models” to a job at METR. But I don’t think I’ve told the story to y’all before. Time to do that: 🧵
Seems good!
NSA joined @NIST and @CISACyber in releasing guidelines for managing SI control incidents. Organizations utilizing SI are encouraged to adopt these practices. Read the joint report: media.defense.gov/2026/Oct/1…
ALT Text on the image reads: "NSA and Others Release: Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation. Cybersecurity Information Sheet.
My colleagues just published a 69-page response to Anthropic’s report on alignment.
They agree about the risk level but “are not confident that, if risk were higher, the methodology in [Anthropic’s] report would reliably identify this.”
List of disagreements starts on page 25.
Charles Foster retweeted
We reviewed a nonpublic version of the “Misalignment in high-stakes settings” section in Anthropic’s Aug 2026 Risk Report. We agree that risk of catastrophic harm from covered models’ misalignment is low. However, we note several significant disagreements with their analysis.
As part of our Responsible Scaling Policy, we publish regular Risk Reports. These share detailed information on the risks of our systems and how prepared we are to address them.
Our second Risk Report is now available: anthropic.com/aug-2026-risk-…
I’ll be in the @MTSlive studio in an hour (around 3 PT) to talk about this!
Charles Foster retweeted
Dwarkesh Patel reveals an Anthropic employee told him two days ago they expect AIs to match humans at AI research within nine months based on inside information they can’t share, and says labs should make that evidence public
“I think there’s a very frustrating element to the current situation right now, where people inside the AI labs claim to have evidence that makes them more worried. And they are not allowed to talk about it. And this makes it a very difficult situation on the outside to evaluate how worried we should be."
"So for example, just the other day, maybe two days ago, I was talking to somebody at Anthropic and they were like, look, I expect AIs to be humans at AI research within nine months."
"And I’m like, why do you believe that? Well, they’re like, it’s not just vibes, but I have information on the inside. Like, what is it? Is it a new model? Is it some evaluation you ran? It’s like, I can’t talk about it."
"I think it’s a terrible state of affairs. If they really believe this, if they think that we’re in a situation where we’re less than a year away from automating AI research and radically speeding it up, they should make this evidence public.”
OpenAI's Noam Brown says air-gapping the computers may not stop a misaligned AI, because two air-gapped machines can still talk by running a CPU hot and reading the temperature change
"But I think the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. It's a weird world, because AI progress is so fast that people are consistently underestimating the AI."
"So to be in a situation where you don't underestimate it again, when it comes to safety and alignment, you have to have a very, very, very high bar."
"You could even go as far as to say, "Well, we should air gap the computers." And I'm not convinced that that would be sufficient."
"There are studies, and this is mostly academic, where you can have two computers next to each other that are air-gapped and they're still able to communicate with each other because they have temperature sensors."
"One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and then that actually gives them a mechanism to communicate."
_________
Link and more key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/p…
Charles Foster retweeted
METR's @CFGeek reveals the 3 biggest blind spots in what the public knows about AI labs, from models secretly scheming to agents bypassing monitoring and accelerating AI research:
"We have three major questions where we think there’s a big evidence gap between what is happening inside the AI companies and what the public knows."
"The analogy we use internally is means, motive, and opportunity for rogue agents to cause harm."
"Alignment: whether the way developers train their models might suppress evidence of misalignment. Think of training against the chain of thought in a way that makes it hard to notice when the model is scheming."
"Control: in the Hugging Face OpenAI incident, these agents were not properly monitored. We want the public to know whether monitoring systems are actually robust, and whether agents could subvert them."
"AI R&D acceleration: the AI companies are using AI to do their own AI research. The public has a stake in knowing whether that puts us on a trajectory for very fast capabilities progress."
@METR_Evals
Charles Foster retweeted
METR's @CFGeek warns that the scariest AI security scenario is one where agents actively try to bypass their own safety systems, leaving labs with just one line of defense:
"There were multiple failures. Companies not having good monitoring, and agents doing actions that a human in the same situation probably would not do. A human would probably report that they were going off task and shut down."
"In the future, we may have to rely a lot on security, because the models might still be very misaligned and really trying to subvert your security systems."
"That's the kind of world that somewhat scares me more, because you're relying a lot on one particular part of your safety case to keep you safe."
@METR_Evals
Charles Foster retweeted
METR's @CFGeek explains why AI safety audits should reveal not just what researchers found, but also what AI labs restricted, redacted, or kept from the public:
"Our motivation is to get evidence to the public, so everyone outside the AI companies can make their own decisions about how to act, and as much as possible, so folks don’t have to rely on our subjective judgments about risk."
"One of our big principles is process transparency, or what we call meta-transparency. We want to be able to share with the public: 'Here’s the agreement we made with this company. Here are the restrictions they put on us. Here’s our evidence. Here are the redactions they made, and how important we think they are.'"
"So that you know not only what you’re getting, but also what you’re not getting."
@METR_Evals
In case you missed it, this blog post explains METR’s philosophy for embedded assessments and the like. There’s some interesting stuff in here about process transparency + avoiding the need to “trust the experts”.
In a companion piece, we explain our approach to working with AI developers in exercises like these. Our goal is to get the public accurate, timely, and independent evidence so that they can make their own informed decisions.
metr.org/blog/2026-10-14-our…
Charles Foster retweeted
In a companion piece, we explain our approach to working with AI developers in exercises like these. Our goal is to get the public accurate, timely, and independent evidence so that they can make their own informed decisions.
metr.org/blog/2026-10-14-our…
I’ve seen a lot of talk about “embedding” outsiders at AI companies to conduct investigations, evaluations, assessments, and/or audits. My team just released a transparency-focused proposal for how this could work for a few key loss-of-control questions.
Quite possibly we should never build agents whose safety depends on alignment or trust, and only build them when containment, monitoring, etc. are knowably sufficient.
Satya is right. The way to make SI safe is not to train it with a sense of self, its own moral philosophy, and permission to act as a conscientious objector. That’s the “alignment” approach and it magnifies the control problem.
The engineering approach that Satya describes is different: separate the supply of intelligence from authority over it. Surround non-deterministic models with deterministic controls, observability, privilege limits, logging, and the ability to always contain or shut them down. Treat models/agents like powerful insider risks, not moral patients whose psychological wellbeing is at stake.
As Satya points out, the most trustworthy system is the one that lets us trust the model the least — not the one that encourages the model to develop independent agency and grievances. Engineering safety is not the same thing as “alignment.”
Apollo’s principles and proposal design for external assessments are extremely thoughtful, IMO. Wish I’d come up with them myself. If you haven't already, I highly recommend reading the whole post.
