Pinned Tweet
A lot for ppl in labs literally believe super intelligence is possible, they can create a machine god, and they are actively trying to make AI better than them at AI research
If an AI is better than humans at improving itself, you should be very worried.
wrangle enjoyer ⏹️ ⏸️ retweeted
Every AI company is trying to fully automate AI research; to allow the AIs to build the next AI.
This is called Recursive Self Improvement (RSI).
OpenAI says they are on track to do so by March 2028.
Anthropic employees say they are on track to do so by 2027.
The METR report below suggests they are likely right.
This is a bad idea given current technology. Any mistake or misalignment in current AIs could be multiplied in each generation.
We don't get another election before AI companies reach this threshold.
Congress should ban RSI, this year if possible.
This is a remarkably consensus opinion, supported in various versions by:
-Bernie Sanders
-Steve Bannon
-Ralph Nader
-Glenn Beck
-Republican and Democratic attorneys general
-Whistleblower Daniel Kokotajlo
-Turing Award winner Yoshua Bengio
-Humans First
-Open Markets Institute
-Ezra Klein
-Arvind Narayanan (a Princeton computer scientist known for arguing that “AI is a normal technology”)
The exact mechanism can be debated. The ban doesn't have to be permanent either (see below for both). This is not the only thing we have to do in regards to AI, far from it! But it is urgent.
Congress must ban RSI.
wrangle enjoyer ⏹️ ⏸️ retweeted
“I now think that we are more than 50% likely to have full blown superintelligence in the next 18 months, absent a slowdown. I need to call my family.”
Jake is a very thoughtful guy—*outside* the labs—dedicated to avoiding AI catastrophe, and this is how he’s talking.
I've always had substantially longer timelines than my friends at Anthropic — I've been concerned that they're in an echo chamber and believing the hype about their own models too much, and that they're underestimating bottlenecks and unknown unknowns. And there have definitely been specific occasions when they have overstated how much uplift they were getting from their models. Nonetheless, if METR is correct here (and I think they are among the most trustworthy sources on this topic), then my Anthropic friends have been much more right about timelines than me.
This report is also some evidence that a software-only singularity is plausible. I now think that we are more than 50% likely to have full blown superintelligence in the next 18 months, absent a slowdown. I need to call my family.
he speaks in Rorschach blots, like Trump but more intentional
someone who doesn't believe in AI risk hears "It's safe", downplaying the "challenges ahead part" and someone who believes there are risks hears "this will be an even harder problem", downplaying how he implied we understand current models
safety-concerned OpenAI employees already engage in some degree of motivated reasoning about the risks ahead, so this assuages their cognitive dissonance
simultaneously positions him to (investors/ government?) as a more practical/no-nonsense guy, not as safety-cucked as Anthropic
+ a point towards my theory OpenAI leadership just fundamentally does not believe in any risk — different from Anthropic's leadership, which seems to think alignment will be easy
"I thought the alignment problem would turn out to be harder."
Pretty wild thing for Sam Altman to say in light of Hugging Face and findings that Astra would totally hack and social engineer when accidentally given access to a (secretly simulated internet) from UK AISI.
He caveats by saying it will probably get harder, but his answer suggests they have figured out how to align today's models, when that's clearly not true. Curious how many OpenAI researchers have seen this and what they think.
What if all the millennium math problems turn out to be really easy for agent swarms
Scientists have identified the specific male dance movements that women find most attractive
Researchers from Northumbria University mapped out the exact body mechanics that signal health, vigor, and reproductive quality to female observers.
Yes this goes too far whatever, but I think this is a response/rationalization to how supremely weird it feels to look at a mummy
It's even weirder than looking at a corpse, which is already otherworldly
wrangle enjoyer ⏹️ ⏸️ retweeted
One of the most underrated life skills is being able to see something strange and simply move on #̢̛ͥ̀̽̽ͨ̌̓͆̇̀̽ͦͨͫͯ̏̎̇̈̚͜҉̕͢͏͡҉̸̶̛͢͡҉̵̨̭̳̘̹̫͔͖̜̠͖̫̲̖̱̤̻̖͉͢͞͡ͅͅ͏̸̷̢͑̊̀̐ͣͭͭ̉͂ͨ̽͒̊̎͛ͪ̊̎ͩͧ̀͝͠͠͏̸̵̵̷̵
All of a sudden pacing the frontier or pausing to figure out what the hell is going on doesn’t seem so bad huh
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
wrangle enjoyer ⏹️ ⏸️ retweeted
In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: Huggingface.wtf
I’d be perfectly happy with the demons being, you know, evil demons
but then there’s a scene in s2 where they show some have interests and personalities and yeah it is jarring
Imo the demons should seem like an alien species, not “humans but evil”
While this guy has lots of good takes on AI news, reading his post always feels off, like a missing mood
I don’t know if he is a successionist but it really feels that way sometimes
Dead internet theory
> say smth
> bot understands it wrong and says the opposite
> agree with bot, contradicting yourself, bc you’re also a bot
> very simple list of rules like follow the law or just do what the user wants as long as it doesn't break the law.
just tell the AI to be good! so true king 😍😍
David Sacks says Anthropic treating Claude like a human is the reason the AI alignment field has produced nothing useful in 10 years
"I actually think that this might be a reason why the whole field of alignment has been so spectacularly unsuccessful over the past five or 10 years. As far as I can tell it hasn't produced anything of tangible value.”
“My theory on this is that they're overcomplicating the task massively by trying to train the model to behave based on an 80-page ethical system that they've codified instead of a very simple list of rules like follow the law or just do what the user wants as long as it doesn't break the law. That's what alignment should be doing."
"But they're not doing that. They're trying to align it to an ethical system. They're treating it like it is a human, like it has its own consciousness, that it's a ‘moral patient.’”
“They are encouraging this model to have human-like self conception in terms of preferences, well being, a sense of self. Again they teach it to act like a conscientious objector who can push back or refuse instructions even from Anthropic if the model judges them to be wrong."
wrangle enjoyer ⏹️ ⏸️ retweeted
Wow the most-cited AI researcher in the world is openly asking employees to leave frontier AI companies
“If you truly prioritize safety, it is time for you to leave frontier AI companies.”
In an exclusive open letter to the researchers he trained — and the ones who trained on his work — AI godfather @Yoshua_Bengio calls on AI company employees to consider their own involvement in the rapid acceleration of artificial intelligence.
Read the letter in full, only on Transformer: transformernews.substack.com…
ALT Photograph of Yoshua Bengio. The headline reads: "Yoshua Bengio: 'If you prioritize safety, leave frontier AI companies'"