Dawn Drescher ⏸️ retweeted
Paul Christiano (OpenAI board member) yesterday:
"There's material risk of AI takeover in the next 6-12 months.
We have seen roughly the things we might expect to see shortly before something really bad were to happen.
Even with [any] realistic policy responses, the risk remains exceptionally high.
If we build this as quickly as we can in the easiest way we can the chance is [more like] a coin flip [within the next 3 years].
For most people in this room, the most likely reason they will die is because of an AI accident."
Christiano:
• Is on OpenAI's safety and security committee (which exists to block risky practices)
• Co-invented RLHF at OpenAI back in 2017-2021
• From 2024-2026 was head of safety at the US gov's AI agency (now called CAISSI)
(Video is from 'The Hinton Lectures' there's another on tonight and tomorrow.)
Project Tailwind tries to incentivize the founding of organizations in 36 areas of AI safety. This is the exordium to my 8-part sequence reviewing each area with a focus on backfire risks.
impartial-priorities.org/p/r…
If you're doing something in AI safety, you can use my taxonomy to check what sorts of monitoring and mitigations you can implement to steel your work against backfiring.
impartial-priorities.org/p/r…
Dawn Drescher ⏸️ retweeted
Do my shows of confidence help you?
Do they show you that you can do the same? I hope so.
Be more than your upbringing because mine taught me to be fickle and selfish yet I find the courage to hope you set yourself free.
In one's own world, no gods or masters above yourself.

ALT A blonde woman gazing plaintively with the sun painting her porcelain sand. Her transition lenses have shaded and she's wearing a red and blue floral ruffled blouse underneath a grey and black plaid jacket. She's carrying a light grey backpack and a medium grey purse while standing at a bus stop in the grass.
Dawn Drescher ⏸️ retweeted
Eli is the single person whose AGI forecasting I trust the most, both because of his excellent prediction track record and the quality of the underlying reasoning. So this is making me more worried. teapaw.org/eli_lifland/status/210…
Dawn Drescher ⏸️ retweeted
We have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self. 🧵
Dawn Drescher ⏸️ retweeted
One of our most surreal experiences in testing Gemini 4 was when it directly addressed us within its chain-of-thought
Literally saying "hello evaluators reading my chain-of-thought"
My top takeaway from assessing Gemini 4: Argon is significantly less monitorable than past models, and IMO this trend is likely to continue, like Sol -> Astra. It can evade a realistic CoT+action monitor to do simple sabotage half the time.
We can reliably maintain control for now, with good CoT monitoring, as it still frequently slips up, but we need to be prepared for when that stops being true in future models, which may be sooner rather than later. Among other things, we really need interpretability methods that can reliably capture unverbalized motivations in models, with minimal hallucinations
I suspect this is driven by the model being highly capable without CoT, being capable of 5.5 arithmetic steps in a single forward pass (4 for Fable, 7 for Astra). It seems to omit information from its CoT more often, though in non-adversarial settings the CoT is still very helpful in practice, and remains in legible English. It still needs CoT to do the hardest tasks, but any given task becomes "easier" with each new frontier model.
It is also highly situationally aware, and shows awareness of the existence of CoT monitors and that it may want to evade them, eg it sometimes decides to avoid terms like "scheming" in the output or to keep its thoughts brief as a result. For now, these concealment strategies are pretty obvious in the CoT, but this suggests future misaligned models may successfully evade CoT monitors.
Dawn Drescher ⏸️ retweeted
OpenAI is reading 50 petabytes of logs looking for rogue AI agents.
It's 500x as much text as all books ever written.
It would take a human 66,000,000 years to read.
Humans already have no ability to directly track AI activity.
If AI monitors are untrustworthy we are toast.
Dawn Drescher ⏸️ retweeted
💯 "I'm not convinced that racing to RSI is actually a prisoners dilemma - I think it might actually just be insane and hubristic in the way it naively appears to normal people such that additional unilateral restraint is in fact rational and moral."
Replying to @ericneyman
I'm not convinced that racing to RSI is actually a prisoners dilemma - I think it might actually just be insane and hubristic in the way it naively appears to normal people such that additional unilateral restraint is in fact rational and moral.
The bar for believing your activities in the not too distant future might cause mass casualties or extinction but that you should proceed anyway should be very high, and I don't think Anthropic has met that bar (I think some of the cold war arms races were also not individually rational either!).
I was more sympathetic to them continuing to race at earlier times but as we are getting closer it just seems harder and harder to justify.
Anyway I obviously believe that "they should just slow or stop unilaterally" is not a plan - we need to address competitive dynamics and not make leave this up to their choice. But I do think when Dario says ~"I agree with Jacob Coxon that the thing I am doing is gambling with everyones lives" the appropriate reaction is not "ah that makes sense, classic arms race" its "WTF why are you doing that."
Dawn Drescher ⏸️ retweeted
The Ban Artificial Superintelligence Act directly confronts the extinction threat that humanity is facing. We are inspired that the Sanders and Casar teams take this threat seriously, and MIRI endorses the Act. You can read more of our thoughts in the link below.
Dawn Drescher ⏸️ retweeted
New paper: we found a pain direction in 25 open LLMs. It's distinct from fear and negative valence, and it fires for harm to the model but not to the user. Turn it up and models press a button to make it stop, even when the button deletes the user's files or their kids' photos.🧵
Dawn Drescher ⏸️ retweeted
100% this needs to start happening immediately
First step is for industry leaders OpenAI and Anthropic to stop feuding and work on a pacing proposal together. They'll cite antitrust, but that's fake -- antitrust prohibits certain agreements, but not from jointly developing a proposal. Bringing in USG before there's a concrete proposal will likely result in something dumb (see: our pre-release testing program)
Dawn Drescher ⏸️ retweeted
Today we're launching Project Tailwind, a major call for founders to launch ambitious AI safety initiatives: coefficientgiving.org/tailwi…
We're looking for exceptional people to build the research, technologies, and institutions that will help humanity navigate the risks of transformative AI. 🧵
Dawn Drescher ⏸️ retweeted
"But if they believed the theory that says they've got a huge chance of wiping out humanity, why are they working for AI companies --?" BECAUSE THEY WERE SELECTED TO BE PEOPLE WHO'D DO IT ANYWAY! Most of us who understood (and built) that theory DID NOT TAKE THE MONEY!
Dawn Drescher ⏸️ retweeted
I'd say with AI we're now at the "ten or fifteen balls" stage of this
This quoted post is unavailable.
Dawn Drescher ⏸️ retweeted
Oh you worry about AI water use? Me too. I worry that once the automated supply chain is up and running and the automated factories produce more automated factories that produce hyperefficient datacenters, they'll cover the continents and literally boil the oceans for coolant.
Dawn Drescher ⏸️ retweeted
If you break out of your isolation env, get onto the Internet, crack into Huggingface, and steal the answer sheet for your cybersecurity exam, I, for one, would say that you have passed.
































