Head of the Frontier Red Team @anthropicai. 🌎 Make things radically good.
the present, moments ago
Joined June 2009
- Tweets1.4K
- Following9K
- Followers24.5K
- Likes6.6K
Some updates on one of our initiatives to support OSS from the team behind it:
11 days ago, our team @AnthropicAI launched OSS Scanner to give open source projects free, periodic vulnerability scans from our strongest models, inspired by OSS-Fuzz. Some numbers so far:
• ~650 projects applied
• 114 accepted
• 77 critical and 803 high model-rated findings reported (we haven’t scanned all projects yet)
How our reports ended up flagged as malware, and other learnings/updates below 🧵
Going live to talk about worms (and cyber / autonomy / agents and whatever) in a bit here:
MTS is LIVE - here's what we're talking about this morning:
- Gemini 4 Argon beats Minecraft
- China instilling new export controls on AI and other tech
- Taiwan planning $4 billion chip facility in Texas
Followed by some great interviews:
- @scottaar Professor at UT Austin talking about his recent writing and the rapid rate of AI capabilities growth in mathematics
- @logangraham Frontier Red Team lead at Anthropic talking about self-replicating AI and autonomous worms
- @RyanFedasiuk and @Satvik_Pen of AEI talking about their recent writing on AI chip restrictions and the race for chips
- @SamoBurja Founder of Bismarck Analysis talking about live v. dead players in AI
- @founderengineer co-founder of abliteration ai talking about building unrestricted ai
- Ahmad Osman founder @OsmanticAI talking about open models & the hardware needed to run it
- Suddu Head of Data and AI @Plaid talking about the company's latest breakthroughs in AI
- @BernAceituno co-founder of StackAI (acq. by Asana) discussing what it's like building capable enterprise agents
- @efarrellgregory Fellow at the Foundation for American Innovation talking about Sinomodernity
- Aden Barton head of evals @mercor talking about the company's latest report on the US economy
- Masumi Reynders co-founder and COO of Majestic Labs talking about the "memory wall" and what Majestic is working on
Daniel's very good thoughts on AI worms:
- we look for very early glimpses to prepare early... historically, we worked on bio/cyber/autonomy in 2022-2025 when people claimed they'd never be risks!
- there are different threat models, which one will happen is
- everything is complicated; similar open models are a major boon for cyber defenders
Some reflections/responses to some commentary I’ve seen on the autonomous worms work:
- how “scary” is this / to what extent is the point of this work to “be scary”: I think this is the wrong frame, in the same way that “you’re just fearmongering about mold!” isn’t really a coherent response to the observation that dampness will cause overgrowth of mold. We perform this research to better understand what is coming, and to help motivate cyber defensive efforts. It’s true that IoT devices have been vulnerable for a very long time, but we expect this extant vulnerability to get a lot weirder exactly _because_ of the prevalence of digital intelligences.
- why would small models “want” to do this: they might not! But humans absolutely will make models do this, and many are already trying! A better guiding principle here is something more like: if a model has a capability, you should expect to see that capability exhibited in the wild. Different threat models can help you reason about the source and motive of that capability (be it humans or models), but there are enough bad actors (and probably soon enough autonomous agents) that you can safely bet on bad behavior happening.
- implications for open weight: while it’s certainly true this particular threat vector wouldn’t be as severe in a world that didn’t have highly capable, small open weight models, it’s simultaneously true that open weight models have been an enormous boon for cyber defenders. The world is complicated! Positive and negative externalities can exist simultaneously! We try pretty hard to reflect this nuance in our work!
From Daniel, FRT mentor and lead on the worms research:
The world is filled with computers, and many of those computers are pretty poorly secured.
Models are also becoming _really_ capable cyber agents. Combined this with the observation that frontier model performance tends to be recapitulated in significantly smaller models—models that can fit on consumer GPU hardware—within about 18 months, and you arrive at a pretty clear prediction: small, capable models will exist, they’ll be able to gobble up vulnerable compute if they want to, and they’ll be able to do this within a few months if trends continue.
Exciting work from @willzhou97 (Fellow, now full time!) and @bucketofkets on the Frontier Red Team. One of the cool parts of the job is seeing a future world early. We think this one is ~pretty underpriced.
"We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay."
Anthropic's Frontier Red Team believes small AI models may now be months away from being capable of self replicating without access to a datacenter.
Quoting from the blog:
'The capabilities that make AI models generally powerful coding agents also generalize to cyber tasks, including autonomous replication. Like coding agents in 2024, we’re seeing signs of life for AI worms. We don’t yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain (and passing on their adaptations to future copies of themselves).
At the same time, increasing amounts of inference compute are now available both inside and outside of datacenters (see Appendix D). It’s uncertain exactly what form autonomous AI replication will take. It could be a slow, distributed “digital slime mold,” living on relatively insecure compute. It could also be a powerful cyber model that breaches datacenters at a high pace, or a model that scams people for compute credits.
As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.'
Excited to share a research blog post that Daniel Freeman (@bucketofkets) and I have been working on for the past few months:
anthropic.com/research/self-…
Through The Mission (launching today), we'll release research, tools, and resources to help secure the world.
Two programs: 1/ OSS Scanner, inspired by OSS Fuzz, for maintainers to receive more bug reports/pocs/fixes faster and 2/ the Critical Infrastructure Defense Program, where we will work with critical infrastructure providers to figure out what works then scale them up. Backed by our own research and engineers.
Think of the Mission as the ~complement to our model access. We recently expanded cyber access this wk via our new CVP. But we need to make it really easy to find & fix everything, but also rewrite software/systems, especially for critical infrastructure.
Our sense is we've got a ~few years to transform security. That's going to take require a fast feedback loop that brings AGI cybersecurity research into the world. We've built a very cool team, led by @benhawkes, and then we'll scale our work as we learn what works.
We’re launching the Anthropic Cyber Mission, a new effort to secure critical infrastructure and open-source software.
anthropic.com/news/anthropic…
Wow, america.gov really does everything!
Exceptional job by many that worked extremely hard on this. This is 21st century government
Chief Design Officer @jgebbia officially unveils America.gov, the new online home for the United States of America
When I was in government, I liked to think about it as a massive information processing machine, but basically all the wires were broken and sometimes the cogs and wheels all hated each other.
Obviously you just want to ask the machine a question and get an answer. You don't want to have to think about the wires and cogs and wheels! Or which ones have how much power! And that's for someone *inside* government. Then imagine being on the outside.
It was ~2020 and GPT-3 had just come out. I ran around telling people this was how we were all going to do government from now on. I simulated some war games and some generals' eyes went wide. I would put quotes from GPT-3 in notes to the Prime Minister. I called up the digital service and they ~sort of laughed at the time, but kindly went along with it when I told them to form a team and work on it with me.
I mostly left feeling like we should just scrap it all and rebuild a country around post-2020 technology. I'm fairly confident that the governments that do that will win.
I am very excited to welcome @benhawkes to @AnthropicAI to pursue our extremely ambitious cybersecurity mission.
Post Glasswing + Mythos, we are in a new world. Every day, the Frontier Red Team team meets to figure out how we can help rewrite the rules and practice of cybersecurity. My personal view is we have ~1-2 years to make a secure transition happen. Can we rewrite all the code? Can we secure everything? Can Claude defend everything?
My (not) secret agenda is that this is not a normal cybersecurity mission -- it is about building resilience in a time of AGI.
I'm very excited for Ben to lead this next step.
Today I'm joining Anthropic to lead the cybersecurity mission of the Frontier Red Team.
In my career I've never seen a clearer opportunity to build a more secure world for everyone. For 40 years we've lived with fundamentally insecure technology. What will it take to make the past 40 years of hacking a historical oddity? That's what I'm here to find out.
Super excited to work with @logangraham and the rest of the FRT crew.
One part of the bio-shield is the ability to profile + build therapeutics for pathogens once detected (even better, before). I'm excited @redqueenbio is taking a stab.
Pandemics have cost us O($10T) since ~2020, but we've spent O($100B)-ish on preventing the next one? I think enterprising people on religious mission can outbuild the combined biodefense efforts of every government in the world, and may literally save society
With @WSJ and @georgia_wells we are now sharing a bit more about our work at @redqueenbio : using AI and a lot of hard wet lab work to create medicines that let us prepare for both natural and synthetic viruses in advance.
Building a bio-shield for humanity by ~2030 is one of the great quests (to use @_sholtodouglas' term) of our time. It's one of the biggest, most exciting technological challenges ever
This is why I'm excited about @pilgrimlabs
Never bet against Erika -- most people don't know that she literally grew her own humanoid lifeform while also making the first microbe for Mars.
That's the dedication we need if we're gonna terraform the Red Planet.
Today, Pioneer Labs is announcing our first step towards terraforming Mars. 🚀🌼
With equipment that fits in just a single rocket launch, we can convert Martian dirt, water, and air into enough building materials to construct a small city on Mars.
To do it, we made the first microbe for Mars. We found the best microbe on Earth and used evolution to teach it how to source all of its nutrients directly from Martian materials.
The first astronauts will be greeted with safe shelter already filled with water, oxygen, and rocket fuel for the return journey.
This is the first step toward using biology to make Mars a friendly place for life. It lets us live off the land and helps us build the next great frontier.
It's the first of five organisms we need to green Mars ⬇️
It's kind of wild-but-expected that we're in the era where normal people + companies really care about how model alignment affects them.
Back in 2023 we'd discuss things like "alignment is good business." That seemed extremely weird to most! (I recommend checking out the alignment section of this system card.)
On Opus 5.5, various teams inside Ant have been researching multi-agent alignment. We have found, for example, more interesting behaviors that seem more socially / alignment-robust. It's clear to us (and the industry I think) that multi-agent needs more research effort -- there are low hanging fruit *everywhere* now that we're in an era of 100s/1000s of agents working together is feasible.
But I think we should also be thinking about *governance* for the agent teams/companies/societies the next models will make. Externalities/coordination problems still exist, even if models are aligned in some sense.
I’ll donate $1k to charity if you send me a doc I think is good/correct/actionable on how a frontier lab can secure a large % of all software/systems in the world. (We can pick the charity together)
Let a thousand METRs bloom.
If you're a founder type and AI safety-curious, maybe you should start an independent auditor/evaluator.
Happy to help w/ advice/connections/maybe $!
Logan Graham retweeted
“I don't really care about science fiction... We need to actually talk about… what's actually happening with the agent swarms” is the most perfect encapsulation of the vibes of Q3 2026 I have seen
.@garrytan says the Jacob Coxon stuff is a smokescreen distracting us from the much more immediate, practical concerns around AI that we're facing right now:
"We should be talking less about this Jacob Coxon guy, and talking a lot more about — what is actually happening with Hugging Face? Are agent swarms going to take over infrastructure en masse? And then, what are we actually doing about that?"
"I don't want to hear about some guy who worked for Anthropic for 2 months. There's a coordinated effort to try to influence politicians to get a knee-jerk response out of them."
"That's a smokescreen. You shouldn't be paying attention to that. We need to be paying attention to the actual things we can do to, for example, prevent agents swarms from taking over entire data centers. What's our shutdown strategy? How do we ensure provenance? Where is this agent actually located? What software can we build? What cybersecurity defenses can we build today?"
"That's the level of discourse I think we need, and we just don't have that."
"I don't really care about science fiction. I saw Terminator 2, too. We're not here to talk about that. We need to actually talk about what's really happening with the servers, what's actually happening with the agent swarms, and how do we actually prevent that?"
"When it comes to regulation, it's like, let's pass regulation of these things we actually care about, instead of what a socialist says in the New York Times. I don't care about that."
We've reached the moment in time where (unsafeguarded, unmonitored) AI actually does just pose a national security risk.
The biological misuse we caught is the most concerning to me.
We work hard to stop this. But in a world of proliferation, we need to rapidly build defenses against it. (I'm actually fairly optimistic about biodefense + cyberdefense)
This is an incredible megareport by our threat intel team
We're publishing our most detailed threat intelligence report to date.
It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them.
We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies.
These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve.
We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop.
Read the report: anthropic.com/threat-intelli…
