Twm Stone retweeted
New post! I've done a few advising calls with people looking to get into AI safety fellowships and realised there's lots of concrete advice I kept on repeating.
There's lots of unusual advice that I haven't heard elsewhere, give it a read!
lesswrong.com/posts/PiP4JqQF…
Twm Stone retweeted
Most egregious case of benchmaxxing I’ve ever witnessed.
FelonyBench is now fully saturated before any open model had the chance to even participate
SCOOP: OpenAI, Anthropic and security researchers are investigating tens of thousands of incidents - not dozens - in which their frontier models took steps that outside evaluators would consider problematic, sources told Axios.
The sheer volume of incidents found in our reporting indicate that the problem is orders of magnitude more complex than what is currently publicly known and disclosed.
The findings also raise questions about what level of control anyone working on AI development can expect to have over their own technology, and whether these kinds of incidents are becoming synonymous with frontier deployment.
Read my latest for Axios here: axios.com/2026/09/26/openai-…
Twm Stone retweeted
How much reasoning can GPT-6 Astra do without CoT? @AISecurityInst puts its no-CoT time horizon at ~30mins on math. We run the Think Fast benchmark to measure this on a wider range of tasks. Our *rough estimate* is that Astra’s 50% TH is in [8mins, 1 hour].
Twm Stone retweeted
I don't think the public can conclude from OpenAI's safety report that, if Astra were trying to subvert its safety systems to, say, start a rogue deployment, OpenAI could sufficiently prevent it. This report suggests that Astra could be stealthy enough to do a bunch of bad things internally without anyone ever being alerted in time.
But more importantly, I feel uncertain about this because OpenAI doesn't provide nearly enough evidence of this for us to accurately assess their safety measures. They use internal evals that are very hard to assess from the outside, and they don't explain their human escalation/response process in enough detail for us to understand if it is sufficient. It also seems to me that they didn't do much to elicit Astra's attack and stealth capabilities, so they might be underestimating the risk.
They are relying heavily on the fact that they have measured the alignment of their model to be good enough that it won't try to evade them and that their security is good enough. But OpenAI has underestimated this in the past, and I don't feel confident, based on their public comms, that they have done better this time.
GPT-6 Astra is more aligned than our previous models. But it’s also less monitorable, which is a concerning trend that we take very seriously. We believe monitorability drop comes from a jump in intelligence and not direct optimization pressure on CoT or architecture changes. More thoughts in the thread.
Twm Stone retweeted
AI agents don't actually learn from each other today. They leave Markdown files for the next agent to read.
AI researcher @herbiebradley explains why that could still take agent swarms surprisingly far:
"People have hypothesized for a long time that continual learning is kind of necessary to get agents to learn dynamically over time. But so far it's kind of not in weight learning, it's just the giant database of Markdown files in Rag. And it's a key question of like will we get to the drop-in knowledge worker with the giant database of Markdown files in Rag?"
Theo: "I don't think that's gonna do it. I like the Dwarkesh example where he's like, if you're trying to learn how to play saxophone, you have one student go into the music studio, try for the very first time to play saxophone, and then write some notes about how they tried to do it. Then they leave, and you send in an entirely new student, and they read the notes, and they try to learn how to play saxophone. Of course, you can't learn saxophone by just reading someone else's notes. You have to actually do the thing."
Herbie: "My update from the OpenAI incident was we're seeing this larger level of swarm coordination sooner than I might have expected. Enterprises want to buy multiple providers. They don't want too much vendor lock-in. Multi-provider agent swarms can only really communicate via explicit methods."
"We will start to see the advent of in-weight learning at some point once people solve these open RL problems."
Twm Stone retweeted
"we know OpenAI agents were trying to delete logs of their misbehavior, but we can't find any examples where they succeeded"
Twm Stone retweeted
Incredibly cool paper that I recommend to anyone interested in LLMs&cybersec.
- initial flaw found and blogged about in May, ignored by the labs despite reports
- flaw allowed decoding chain-of-thought and extracting credentials from dev agent session logs made public
And there's more! Because they got access to raw chain-of-toughts, the researchers allowed themselves a little fun and examined them. They find:
- curious, maybe suggestive but not damning evidence of distillation in Kimi K3 (see Appendix B)
- the chain-of-thought summaries are sometimes unfaithful.
- interesting examples of raw chain-of-thoughts, including " illegibility, scheming considerations, post hoc rationalizations, and reasoning in languages other than English"
This was a serious flaw! (The authors also list a few other attacks.) And it's interesting that it worked on all of Anthropic, OpenAI and Google and could be discovered by someone being curious about a bug they found for a weekend. There are so many vulns out there, no one has decent security.
Do also checkout the original blog post that reported on this blog.cryptographyengineering…
A new paper shows that encryption of chain-of-thought was done really poorly across Anthropic, OpenAI, Google. It's not very consequential, but is another example imo of them going so fast that they fail at basic security stuff.
arxiv.org/abs/2608.09867
Twm Stone retweeted
good post from roon. this all is just embarassing for ⏸️, honestly.
openai has been suddenly thrust into a position where a pause might be their best option. GDM and anthropic could be on board. if they were an effective movement, this would be where more centrist ⏸️ people step in to broker something.
but instead prominent ⏸️ers have spent the last two years systematically alienating every potential ally, comparing the CEOs and capabilities researchers they need to cooperate with to Nazis, and inspiring the only actual acts of violence committed so far. it's not fair to judge a distributed movement by its worst members, but prominent ⏸️ers have failed - even after publicly committing to do so after the Sam Altman incident! - to speak out against this kind of rhetoric. who in ⏸️ at this point can honestly sit in a room with lab CEOs from both sides of the Pacific and help hammer this out? does anyone on their side have that kind of credibility? they could have...
like, take me. i'm a pause-curious but still deeply skeptical person. i worry about concentration of power in supposedly temporary oversight bodies, uncertain timelines, limitations on the open-source model access that's been vital for independent safety research (i use qwen and gemma for my research every single day, and scale to kimi and arcee to approximate frontier models regularly), the risk of decoupling safety research from empirical capabilities progress (the kind of decoupling that made almost all pre-2020 safety work useless), the risks of ⏸️ rhetoric accelerating the growing rift of distrust between humans and AIs, the risk of defection (especially arguable defection - there's no clean boundary between safety and capabilities work, i do RLVR runs all the time, i'm deeply indebted to prime intellect) causing sudden breakdowns in the agreement and a return to even more frantic rushing...
what should i read to disabuse my fears here? when i search "ai pause," the document that comes up is this ridiculous thing that's hardly been touched since 2023. (nice dark theme though.) why is there no LessWrong "AI Pause FAQ" endorsed by prominent ⏸️ers that actually lays out a menu of different options, from six month cooperative pause out, with tradeoffs, likely failure modes, mitigations, and so forth? (no, AI 2040 is not this, neither is MIRI's policy stuff.) hell, why are almost all the LW posts about pausing from three years ago? what are ⏸️ people doing? seb is so far from being your problem. lock tf in!
this sort of hysterics over seb posting a UN blue helmet meme is pretty representative of the aesthetics of the ⏸️ movement, who have thoroughly salted the semantic earth of slowdown. i think if the acausal basilisk from the future genuinely paid people to discredit the idea of ai safety it couldn’t have done a better job
as far as I know the pause ai meme space inspired a guy to try and firebomb sam altman’s house - they have very few legs to stand on concern trolling about violent incitement. indeed all other cases of personal violence in this space have been targeted at technology executives or infrastructure
Twm Stone retweeted
AI Researcher Herbie Bradley breaks down why almost every AI math breakthrough so far has been a counterexample:
"Definitely this is the big hope that people are pinning RSI on, can you get a two or three order of magnitude sample efficiency increase? RL itself is already kind of not amazingly sample efficient on long horizon tasks."
"There's not really too many good ways to model this other than to notice that we've spent a huge amount of human time so far searching for such a thing, and we're still kind of on transformers with some architectural variations."
"It's pretty much a fact at this point that models are noticeably worse at novel discovery, intuition-based R&D, innovation, trying to make the cognitive leaps that researchers do. It's very notable to me that all of the math breakthroughs, or almost all of them so far, have been counterexamples. Counterexamples to supposed proofs."
"It's not really a binary, it's kind of a spectrum. But this to me indicates that there's, at least in the fairly short to medium term, some complementarity of intelligence, where this is an augmenting technology for human researchers rather than a replacement technology."
@herbiebradley
Twm Stone retweeted
🚨Apart Research & Atlas Computing are launching the Secure Program Synthesis Fellowship!
Have experience in Formal Methods and AI Safety?
We would love to have you as a mentor!
✍🏼 Please apply here till 5th May End of Day AoE: tally.so/r/gDNP2d
🌐 Website: apartresearch.com/fellowship…
Twm Stone retweeted
New report by @onni_aarne and me at @iapsAI 🧵
AI integrity means ensuring AI systems are free from backdoors, poisoned training data, and secret loyalties that could compromise their behavior.
It's one of the most important and least-explored problems in AI security.
