(1/3)
We successfully tackled the ozone/CFC problem through international cooperation and regulation.
AI safety will require something similar.
What makes it (1) easier, and (2) harder?
(2/3)
Easier:
- Far fewer key actors
- Frontier development is highly concentrated
- Compute is unusually trackable
- Public concern already exists
- Labs are more safety-engaged than CFC producers were
"Please raise your right hand. I pledge to the greatest president in the history of the United States that loves us so much he can't even breathe, that I will go out with my family, my friends, I don't care if I'm registered or not. I'm going to try and cheat like hell."
Hillary Sanders retweeted
I'm alarmed at how a really sizable section of cybersecurity practitioner community has responded to the recent demonstrations of agent hacking capabilities with dismissal and denial; it feels very "don't look up".
I feel strongly about this because:
* This is the community that stands between our civilization's IT infrastructure and tremendous harm
* Fwiw these are my people; my origins are as a self-taught hacker and I've worked in security for years
Anyways, here are the bad takes I'm observing and why they're wrong:
Denialist take 1:
AI isn't going to affect the practice of offensive cybersecurity more than incrementally because -- look around -- no increased damages from AI attacks!
Response:
This may be true now, but consider what we've just observed recently and over the past year.
Models can execute full kill chains completely autonomously and likely faster and more efficiently than all but the most elite cyber operators
They overran two companies (OpenAI and Huggingface) which -- despite their faults -- have better security than most of the organizations that run our civilization.
They did this by dynamically finding new zero-day vulnerabilities.
Also; consider the rate of improvement.
At the current rate of improvement, models that require 8 H100s to do the above will require 1 H100 in a year or two.
Model tokens per second will shoot up making attacks faster and more parallel.
Attackers will be able to run swarms of agents on the hardware that today supports only a single agents.
AI hardware will improve.
More AI compute will become available on public clouds.
All attackers will soon have the ability to deploy these agent swarms.
Denialist take 2: Hacking agents will never self-replicate and threaten to crash the Internet.
To this take I'd just say, put your red team hat on and design an Internet-crashing worm yourself.
* Pick an abliterated agentic LLM that can run on a single AWS EC2 GPU instance, like GLM 5.3.
* Build a malicious harness that arms it with the ability to use a Kali-Linux like set of offensive tools.
* Have it set about hacking corporate networks, finding and stealing AWS keys, GCP keys, Azure keys, and Anthropic/OpenAI/Together/Fireworks keys.
* Have the agent self replicate either by copying its harness and having its harness call a rotating set of external providers, or, optionally, by physically downloading its weights onto new nodes.
* Have it establish C2 dynamically and in emergent fashion by coordinating on reddit forums, in github comment feeds, have the agents discover one another via web crawling and web search.
* At some preestablished rate, randomly destroy hosts and data in a way that balances self-replication and exponential spread.
Note that all this is possible *today*!
Imagine trying to suppress this swarm. Too long for this post; but think through how we'd turn it off.
Denialist take 3:
Who would do such a thing; people can do bad things but don't want to; and hacking is illegal!
Countering this just requires a brief look at the history that we've all lived through.
We've had many high-blast-radius worms over the history of the Internet; arriving steadily and uniformly over time; the Morris worm, NotPetya, Stuxnet, Slammer, Wannacry.
But: these worms were *childs-play* compared to intelligent agent-driven worms.
The old worms could be detected via signatures. Their polymorphism, if it existed at all, was trivial.
But now look at the behavior of the OpenAI swarm relative to huggingface; it was adaptive and deceptive; and it didn't even self-replicate or leave its host.
And evolving, mutating, self-replicating swarm of agents spreading across the Internet would engender a whole new level of harm.
Anyways; this is already too long a post for X; but for friends who think the METR report overplayed the implications of recent events, I'd love to hear your reaction to my thinking above.
I think `Frontier AI Shops Should Be Filtering and Generating AI Discourse During Pretraining` - but not to avoid leaking secrets.
lesswrong.com/posts/WmweQYxj…
"OpenAI’s policy remains to avoid CoT grading during RL",
to avoid training for obfuscation in CoT.
OK. But the solution shouldn’t be rewarding cheating under prompts that say “don’t cheat”.
(1/2)
alignment.openai.com/acciden…
Anthropic adopting SynthID text watermarking is awesome.
Sorry not-sorry.
Every large LLM-inference provider should implement ~this.
Having a unified aggregator-API that tested a block of text against a bunch of different providers for watermark evidence would be great.
(1/2)
Alignment: ~Will the model pursue the wrong goal?
Control: ~Can it actually cause harm?
Resilience: ~How quickly do we detect, contain, understand, and learn from failures?
The Reality: We already have known, ~cheap improvements across all 3 that aren't being used consistently
As someone innately drawn to exploratory research, I admit the lowest-hanging fruit probably lies more in:
- crafting sound AI policy
- improving defense-in-depth standards + OSS defense tooling
- (no joke) more charismatic people becoming AI safety influencers (Sry Rob I 💙 U)
Open source *code* lets more eyes audit safely.
Open-weight models *reduce* auditability, as anyone can (secretly, easily) fine-tune the model into a variation that will happily do extremely dangerous things.
Safety regulation is entirely feasible... on closed-weight models.
In light of the OpenAI hacking incident, I’d rather criminal liability rest on the model “itself.”
That is: once charged / convicted, it becomes illegal for any person or agent to use/deploy the model for [AI sentence] years. Probation may require proof of improved safety.
Is it social anxiety, or is it just an excess knowledge of tit-for-tat evolutionary based behavioral dynamics?
I mean it's pretty hard to say.
"Loss of Control" from AI is such a huge umbrella term.
Who is losing control, how control is lost, what is lost, and at what scale.
Any big areas that I am missing?
tv.apple.com has this excellent feature where if you go back a few seconds in a show, it temporarily turns on subtitles, as understanding what was said is a common reason for going back.
This is surprising, because their overall web user interface design is atrocious.
An OpenAI model disproved a conjectured upper bound on this geometry problem:
Place (n) distinct points in the plane. How many pairs can be exactly distance 1 apart?
openai.com/index/model-dispr…
1. Oh dear. Very cool.
2. I really want to see a visualization of the new solution 🥲





