We study the strategic capabilities and motivations of AI agents.

Joined May 2024
Palisade interviewed 22 current and former employees from OpenAI, DeepMind, and Anthropic about their personal views and fears around AI development. Today, we’re releasing the first batch of those interviews. Please watch and share.
46
254
66
1,287
271,958
New self-replication research from Anthropic on even smaller models! In May, we found that Qwen3.6-27B could sometimes self-replicate by exploiting vulnerabilities across a network. Now it seems we're down to 9B models
Anthropic's Frontier Red Team believes small AI models may now be months away from being capable of self replicating without access to a datacenter. Quoting from the blog: 'The capabilities that make AI models generally powerful coding agents also generalize to cyber tasks, including autonomous replication. Like coding agents in 2024, we’re seeing signs of life for AI worms. We don’t yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain (and passing on their adaptations to future copies of themselves). At the same time, increasing amounts of inference compute are now available both inside and outside of datacenters (see Appendix D). It’s uncertain exactly what form autonomous AI replication will take. It could be a slow, distributed “digital slime mold,” living on relatively insecure compute. It could also be a powerful cyber model that breaches datacenters at a high pace, or a model that scams people for compute credits. As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.'
8
7
1
47
6,042
"The Pentagon asked for more funding for AI weapons than for the whole Marine Corps this year." Alex Turner was a researcher at Google DeepMind until June. He left over a deal between Google and the Pentagon that he believes violated Google's principles around AI.
4
6
1
29
2,049
"They're building very powerful systems that they don’t understand, that they can't precisely shape, and that many of the people in the companies know has a significant risk of literally killing everyone." Jeremy Schlatter was at OpenAI back in 2017; he's now a researcher at Palisade.
4
11
36
1,418
Palisade Research retweeted
This is an absurd reason to fire someone, and I do not believe it was the actual reason
Replying to @j_asminewang
OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn't remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden.
10
8
314
14,218
Palisade Research retweeted
Firing the lead company contact for an independent investigation that revealed crucial information about the most important incident in the history of AI is an extremely bad look. OpenAI cannot credibly claim to take these incidents seriously and then behave like this
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us? This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them. I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job. For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired. I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
8
25
272
5,430
Palisade Research retweeted
This is not looking good for OpenAI. When a company is trying to build superintelligence, they should be subject to extremely high standards of transparency and scrutiny. OpenAI has a duty to respond to these claims
Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership. I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
3
7
1
83
2,335
"These models will have an incentive and a propensity to game those tests and get deployed even if they are not aligned." Mary Phuong is an AI researcher currently at Google DeepMind. She expresses deep concerns with developers' ability to judge the safety of their frontier AIs.
4
17
8
57
6,275
Andreas Kirsch left DeepMind last week. The week before that, he told us that pushing the frontier creates risks for everybody: "This entails all current people, this entails our children..."
4
11
62
3,211
From the former Policy Frontiers Team Lead at OpenAI: She worked on testing AIs for dangerous capabilities, and warns we don’t really know how to do that. "The evaluations can't be comprehensive enough… the models can get smart enough to hide their capabilities." Rosie Campbell highlights competitive and cultural pressures she experienced inside a frontier AI company, and discusses her hopes and misgivings about pacing the frontier. "I assumed there would be a whole load of good arguments for why this isn't really something we need to worry about. There'd be answers to all of these concerns. And the more I read into it, the more I was kind of terrified at the lack of rebuttals to some of these arguments." "Over time, I feel like the organization changed a lot. The incentives changed. There was a lot more pressure to move fast to build commercial products. And a lot of the people who had been very motivated by these big picture safety concerns started leaving." "We're at a point where we don't understand what's going on with the systems. The rate of progress is kind of insane, and we need to make sure that our ability - to understand, control, deploy these systems safely - that keeps up with the capabilities of the systems themselves."
6
15
5
74
11,614
10-year DeepMind veteran: "It doesn't have to be sentient or conscious or anything like that to have these instrumental goals... If you have powerful AI controlling the planet, maybe it will cover the surface of the planet in data centers." Victoria Krakovna has been a research scientist at Google DeepMind for almost a decade. In this interview she explains why she’s worried about the trajectory of advanced AI systems, and why she thinks humanity should coordinate on slowing down AGI development. "This is a problem that does not get easier as AI systems become more capable. It becomes harder because more advanced systems can better optimize for the wrong thing." "I think there is a significant possibility that AI development could lead to human extinction… Gradual disempowerment overall looks more likely than extinction." "We'll need to coordinate a slowdown, not necessarily a complete ban on AGI development."
7
19
4
80
4,291
Ex-DeepMind employee Vishal Mani: "The gap between Neanderthals and Homo sapiens in intelligence was far narrower than what we should expect the gap between humans and advanced AI systems to be...The historical precedent is not actually so reassuring." Some things Vishal shared in his interview with us: - "I do believe there's a real risk of human extinction through the AI transition." - "Artificial general intelligence or artificial superintelligence is a starting line, not a finish line. It's the moment when we hand over the baton to AI that can do research and development." - "Pacing the frontier really means going from insanely fast progress to very fast progress." - "If we don't pace, we roll the die." - "The truth is being said out loud because recursive self-improvement is now so imminent that no other option makes sense." - "The obvious answer to winning the race is to go faster, but that's not actually a viable strategy if we lose control." - "If...the weights are not defensible against nation state actors, then in practice you don't actually have a defensible lead."
11
16
4
65
3,384