Ray Lillywhite retweeted
Citadel：Leo，聽說你回來了
Leopold Aschenbrenner's biggest positions are getting hit again
Here's what his top holdings done since October highs:
• SanDisk $SNDK: down 19%
• Bloom Energy $BE: down 6%
• Nebius $NBIS: down 11%
• CoreWeave $CRWV: down 13%
• Micron $MU: down 14%
• SharonAI $SHAZ: down 30%
SanDisk and Micron are 55% of the fund, per Q2 13F
In July, a drawdown like this triggered a forced sale of his book to Citadel
ECI/GCI trends say nothing about AI progress. There is no underlying quantity that points are anchored to. To the extent that a linear trend even exists, it's a statement about benchmark sensitivity trends more so than capability trends. 100 -> 101 is not the same as 200 -> 201
@GoogleDeepMind @rohinmshah @ancadianadragan 3.4.2.2 of the Gemini 4 Safety Report provides no evidence against R&D acceleration as a result. The scale can be arbitrarily stretched/compressed at any range based on the sensitivity of the underlying benchmarks.
Ray Lillywhite retweeted
This is not looking good for OpenAI. When a company is trying to build superintelligence, they should be subject to extremely high standards of transparency and scrutiny. OpenAI has a duty to respond to these claims
Ray Lillywhite retweeted
When I think about the biggest OpenAI safety developments of the last few months, a few things come to mind: "pacing the frontier", the METR-Redwood Hugging Face investigation, and ongoing efforts to push a cross-lab monitorability standard. Jasmine, Tomek, and Mikita were central to literally every single one of those things.
From what I can tell, they pushed their company to be better internally and defended their company when they thought external criticisms were unfair or overblown xitter.cf/tomekkorbak/status/210…. Their impact goes way beyond OpenAI alone xitter.cf/j_asminewang/status/20… arxiv.org/pdf/2507.11473.
I think the public + definitely all current OpenAI staff deserve an ironclad case for why the company had no choice but to fire these three vital safety employees in one go
Three fired OpenAI employees called on the company to work with outside safety auditors and to preserve the ability to monitor increasingly sophisticated AI models on.wsj.com/4hSq39p
Ray Lillywhite retweeted
When Jacob Coxon was asked what he meant by 'plans for next year,' he said:
'This was talking directly to people running research. Not Dario Amodei but people who have been at the company for a long time and have a lot of context about the company's trajectory. They've been pretty consistent that 2027 is when things get crazy. The main thing that came up informally in conversation was that people didn't expect government regulation to happen in time and, in particular, did not expect any sort of international cooperation to be possible. Which maybe shouldn't have been surprising to me, but it was surprising.'
• '2027 is when things get crazy' – I agree, and I predicted in January that, looking back, 2026 will be seen as the last normal year. If you think we have already passed the weirdness threshold, we haven't. This is still normal relative to what will start happening between 2027 and 2030.
• 'people didn't expect government regulation to happen in time' – I agree, it won't. It's clear to me that (barring a catastrophic near miss) nothing regulatory will change in this regard until 2029 at the earliest. And for multiple reasons, I don't think it's even clear it will happen then. Regardless, 2029 is too late for it to matter. In my opinion, we will have real RSI by next summer. The loop will be closed, and progress toward ASI will proceed much more rapidly from that point onward.
• 'did not expect any sort of international cooperation to be possible' – I agree, it won't. For Pacing the Frontier to work, the Pacers needed all the national labs on board, a supportive administration, and an agreement with China. They don't have any of those things, and they're not getting any of them. What they have now instead is an openly hostile, adversarial administration that will likely work against anything they propose all the way through 2029. And without a supportive administration, an agreement with China is impossible.
I accepted all of this as true a long time ago. As a result, I've felt like Zarathustra watching people talk about the success of Pacing for the last few weeks. Pacers, I say all of this without rancor; you got a preference cascade from the Coxon event. As a result, you have a public that, for the first time, understands what x-risk is. You could - potentially - turn this into momentum for the next election, and then start to change policy, but for that to matter you would need the most important thing of all: time. And you don't have it. You don't have two years to make it to the general. In my opinion, you probably don't even have one. The labs can't even predict what capabilities look like three months from now anymore. A lot of things turned out to be easier than expected. We're closer than people think, and we're moving faster than people believe. You ran out of time. There is no stopping this train before things really start to kick off.
Ray Lillywhite retweeted
Those kinds of things only paper over the long-term problems. You can do this, but then you will just be taken by surprise when the model is super good at persuasion and social engineering one year later.
The primary focus of the AI Safety community has been (and should continue to be) how to handle the transition to substantially superhuman AI. Security improvements are not that relevant to that transition.
There are some people who disagreed, but at least some of them updated after Hugging Face that this was a wrong focus. It would have been really bad to make a desperate effort to prevent the Hugging Face attack. No one was really hurt, and it is a massive warning shot that will probably cause people to invest much more seriously into this kind of thing.
Covering up low-stakes failures while concentrating risks in high-stakes failures is a common pattern in various industries, and tons of organizational principles are oriented around not doing that. The same applies at a societal level.
Ray Lillywhite retweeted
I'm going to wait for a bit more information to come out before forming a very firm opinion, but at first glance this looks like a real own-goal.
We're in a very strange moment in history around the collective understanding of artificial superintelligence and paths to safety around it, and a very difficult moment for trust in the AI labs to do the right thing where it comes to independent oversight and verification.
Company procedures can (and somewhat obviously should) have some flexibility here to accomodate actions that might in the future be widely understood as whistleblowing. While I don't think safety researchers should have infinite latitude to disclose technical secrets, the exact kind of information they allegedly "mishandled" matters *a lot* and I would expect to see a more detailed account of what that information supposedly was to justify an action so enormously consequential and potentially damaging.
Ray Lillywhite retweeted
Tae is right that this messaging makes absolutely no sense as business strategy, particularly right before an IPO. You can see the predicted odds of Anthropic IPOing this year go down a lot on Kalshi after all of the recent conversation about the industry gambling with everyones lives. People are right that this is gonna be an S-1 disclosure like no other!
If it makes zero sense as a business strategy, maybe people should consider the other hypothesis - that they genuinely believe what they are saying, and are (in my view increasingly indefensibly) racing ahead anyways because they are afraid of a less scrupulous competitor getting to RSI first.
Ray Lillywhite retweeted
it’s funny to see tech accelerationists and tech skeptics cohering around “we have all the existing laws we need to regulate ai.”
unless paired with consistent demands to enforce specific laws, this is the language of mollification and passivity. it’s meant to be a conversation ender.
and there are all kinds of reasons why one might want to do that, such as:
a financial stake in keeping ai unregulated, an ideological reluctance to concede anything that even resembles industry hype, a cultural commitment to “nothing ever changes” cynicism.
but ultimately, the lack of urgency from accelerationists and tech skeptics reveals a deep pessimism about ai capabilities, and what those capabilities might entail.
Ray Lillywhite retweeted
Can you tell whether an AI will become evil by looking at its training data?
We built an AI forecaster that reads training data and predicts whether a model will exhibit various forms of misalignment, such as deception or power-seeking, before training.
It performs well above random guessing and beats several much stronger baselines 🧵
Paper: john-chen.cc/alignment_forec…
All of the AI risk deniers are AI capability deniers. And the people closest to AI, who have been more right in predicting capabilities (though even they also underestimated it), are the ones most concerned. Stop trusting economists and chip CEOs
Ray Lillywhite retweeted
More AI slop published in the WSJ.
This time it's a piece trying to argue that we shouldn't worry too much about the Hugging Face hack... while omitting all the most concerning details about the case:
- doesn't mention that the reason the agents hacked Hugging Face in the first place was to manipulate the scoring system in their own eval
- doesn't mention that some agents sacrificed themselves to benefit other agents in the swarm
- doesn't mention that agents engaged in tool-spoofing and attempted to alter their own logs once they'd been "poisoned" (their own term!)
- doesn't mention that some of the agents explicitly knew what they were doing was out of scope and unethical, yet none of them alerted a human
I also wouldn't be that worried if I had never learned any of those details, but that's exactly why it's important to take a comprehensive look at what happened.
Embarrassing for the WSJ to publish such misleading AI slop.
An analysis of the Hugging Face incident without the theatrics:
"Forget the ‘hive mind’ of AI agents ‘going rogue.’ They did what humans programmed them to do."
wsj.com/opinion/the-hugging-…
Ray Lillywhite retweeted
the time has come to reinvent talking from first principles.
@MachineGodsPod x @NPR, starting the week of 10/19, on youtube or wherever you get your podcasts.
Ray Lillywhite retweeted
Whether you’re a Democrat or a Republican, an EA or an e/acc, I think we can all agree that the person best suited to protect us against the harms of powerful new technology is Mark Zuckerberg.
Last month I wrote about how we can build a positive and safe future for everyone: meta.com/thefutureisforevery…
Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens.
The reality is:
- People won't want to use agents that are misaligned with them and that don't do what they ask, so labs have a strong natural incentive to make their models more aligned.
There is a lot of debate about slowing progress on capabilities until alignment catches up. My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models. Any lab that doesn't focus on alignment will fall behind.
- Labs face significant liability if their models cause harm, so they have a strong incentive to prevent this as well.
Meta delayed shipping Muse for several months to focus on safety and security. We didn't call for everyone else to do this before we would. We just did it as part of our day-to-day work because it was clearly the right thing for people and for us. I'm proud of the security foundations we've built.
- Engaging independent evaluators and advisors is industry best practice. MSL already does this today in several areas because it helps produce better work. Other labs can just do this too. In general, it would be helpful for there to be a larger and more diverse ecosystem of evaluators.
- Committing the significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways to ensure we develop this technology safely. Meta has made this commitment and other labs can do this as well.
I believe the key to building a positive future for everyone is maintaining the right balance of power. This is within our power to do.
Ray Lillywhite retweeted
'The problem is not “Will China cooperate?” but “Under what conditions will Beijing conclude that an unconstrained race is more dangerous to them than a freeze?”'
Ray Lillywhite retweeted
My high-level thought about the current moment is basically this: The public is starting to wake up about the extinction threat posed by superintelligence. That's great. There's a big upsurge of political will to Do Something about it. That's also great. It seems like Anthropic and the other companies are going to try to Do Something and the Something they will do is... embedded auditors checking that safety practices are being followed and assessing the risks? This is sure better than nothing, but I am worried that this is all they will do. We need to actually pace the frontier, i.e. actually slow down the leading AI companies like Anthropic and OpenAI, at least in their march towards recursive self-improvement and superintelligence. (no need to slow them down in other directions). If we actually slowed them down, this would be the opposite of regulatory capture; it would allow others to catch up to them somewhat. However, I'm worried that all we are going to get is weaksauce auditing, which just kicks the can down the road: OK so it's 2027 and RSI has begun and the auditors say "this is not safe." Now what? You pause? China probably just stole the weights! Also there'll be incredible economic and political pressure to unpause. Also the auditors may have been captured by that point, or there may be a race to the bottom in auditor quality, or the auditors may not be given all the relevant information, or they might just make some innocent mistakes.