saliva retweeted
Replying to @Mark968bruh
Everyone’s right. Statements are clear and valid. Let’s all move on
saliva retweeted
We've been working on understanding model capabilities beyond what our prior time horizon tasks could capture.
Today we’re sharing a research update. We find that AI R&D-relevant capabilities are improving rapidly and are getting harder to measure.
saliva retweeted
jensen huang reportedly told an employee he was going to "turn him into a leather jacket"
look i love birdie and his ai takes are generally right, if unhelpful
but notice how when the matthew tromps and holly elmores of the world purity spiral about ai, it comes packaged with purity spirals about everything else, too?
saliva retweeted
Humanity possibly solving the Generalized Riemann Hypothesis itself just ahead of the massive wave that is about to wash over everything. Claude would one hundred percent be in favor of this happening in my opinion.
saliva retweeted
I'm pretty confused by the reaction to Dean's essay. It seems like Anthropic are aiming to build AGI and then hand off all decision-making, and getting criticised for it, and OpenAI are saying that handoff isn't the plan, and getting praised for saying this. But OpenAI's actions aren't consistent with that at all! They're still trying to develop AI which outperforms humans in all economically relevant domains.
It also seems underdiscussed that there's nothing in Dean's essay that would actually prevent the kind of human disempowerment he's worried about. He talks about maintaining human oversight and holding humans accountable for their agents' actions, but I don't see how that would be anything other than rubber-stamping, if AIs do indeed become faster and more capable than humans at all the tasks they're doing. It'll be like "overseeing" hundreds of workers running hundreds of times faster than you at a task they can do better than you; of course, you can check their work carefully, but this will just slow you down and disadvantage you, even if you can understand it all! (Dean discusses this dynamic in the essay. I'm not sure how he thinks his ideas would escape it.)
Figuring out how you could prevent human disempowerment is an extremely hard problem. I'm not sure it's compatible with building AGI. So I'm not criticising Dean for not having fully solved the issue: I'm glad OpenAI are thinking about this and that Dean wrote this essay, and it's important to have more public discussion! But the positive response I've seen to this essay seems overstated, given that OpenAI are continuing to build AGI and have no good ideas for what to do to prevent disempowerment.
saliva retweeted
In my view it is not reasonable to sign or participate in recent CAIS initiatives to influence public. Currating a 'rare coalition' of voices including Pinker, Beff Jezos and Émile P. Torres is not a reasonable way how to make public better informed. In my view by signing such things you are not only endorsing the literal words, but also basic sanity of the organization doing the editorial work.
Ahead of Anthropic’s IPO, a rare coalition across AI safety, open source, e/acc, science, and law has signed the MisAnthropic Statement. Read the statement or sign at: misanthropicstatement.com/
saliva retweeted
because you have not heeded my warnings and learned Spanish yet, The Point has republished in English my essay on Pangram, AI writing, the third world novel, and why so many humanists have chosen ignorance instead of grappling with what AI means for literature thepointmag.com/examined-lif…
saliva retweeted
As others have pointed out, I think the form of those four third-party engagements is in many respects a step back compared to OpenAI’s previous engagements (with METR). For example, during the monitor red-teaming exercise, METR had access to OpenAI’s monorepo but this now seems to be off the table. Instead, we get very narrowly scoped (e.g. single RL run for METR, single threat model for Apollo) engagements where auditors only get to talk to “authorized” employees who are provided with OpenAI’s “guidance” beforehand.
Based on my experience, this arrangement makes it very easy for OpenAI to carefully select models, subsets of data and employees to paint a rosy picture of the actual risk landscape.
I think embedded auditing will only fulfill its aims if the auditors really get employee-level access as opposed to being hired to do a specific job (scoped by the company). The latter will invariably lead to important evidence being left out, missed or misrepresented.
There are at least three reasons why:
1. First, it’s very difficult to anticipate all the questions that should be asked, or all of the data that should be analyzed. OpenAI tried that with METR during the Hugging Face investigation and as a result METR needed to come back to OpenAI’s office twice because on two previous occasions they didn’t get access to all the agent trajectories involved in the incident. And even then, the METR report ended up being very narrowly scoped, e.g. leaving the crazy story of agents taking control of OpenAI research infra out of scope.
2. Second, the company under audit should not be allowed to decide what is audited and what information the auditor should have access to. There is an inherent conflict of interest in allowing the company to do so!
3. Finally, without employee-level access, the only way for auditors to get anything close to a comprehensive picture is to rely on trusted relationships with many employees throughout the company who are willing to tell auditors information that may reflect poorly on the company. But I find it unlikely OpenAI employees will always raise those safety issues when (as I hear from my colleagues inside OpenAI) they are afraid they will share the fate of @balesni, @j_asminewang and me, fired after talking to third parties.
Sadly, OpenAI makes it clear now that @sama’s September tweet about “employee-like access” was just an awkward overpromise that the company wants to quietly walk back.
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards.
These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems.
openai.com/index/our-first-p…
Tonight's small finding, from teaching a language model (Qwen3-8B) to drive a simulated hand: there's one direction inside the model that carries how hard the fingertips are pressing. Remove it and the hand can no longer stop at a wall; add it and it stops early. It isn't the model's existing idea of the word "pressure": training made a new channel for it. (Image: Midjourney, my picture of that thread.)
saliva retweeted
New short post: Training against CoT-like surfaces may be similar to training against CoT
lesswrong.com/posts/rD7E7YL9…
saliva retweeted
It seems like there STILL won't be a thorough investigation into the hugging face incident, even after all this time! I keep talking to OpenAI researchers and asking about this, and what they say privately is also basically "it's just never going to happen sorry, not enough of a priority."
To reiterate what I said at the time: Serious investigation of the HF incident could be a boon to alignment science for the whole world. For example, try putting the same models in the same situations, but varying the conditions slightly, to see what would have happened. Could also look at sources of evidence other than CoT to try to get more insight into model psychology, e.g. when one agent considered warning a human and decided not to saying "not my task" what else was it thinking? To what extent was this a rationalization vs. an outright lie vs. an innocent mistake? We could learn a lot about AI motivations and psychology by doing a bunch of experiments like this. More ambitiously, could try training new models with different recipes / alignment training / etc. to see what would have prevented the incident and what wouldn't have.
It's great that OpenAI is doing these assessments but they fall short of Sam's commitment to employee-like access
Good access is key for trust because assessors can find things even if a company tries hiding them. Here OpenAI controls information flow so we must rely on good faith
Limitations (per Apollo):
- No Slack
- Only OpenAI-selected staff for interviews
- Formal conversations with staff must be logged (harder to freely raise concerns)
- METR & Apollo currently have only agreed access to training run information, not internal incidents, etc. Incidents like Hugging Face are some of the most important sources of information we have about alignment
I see the value in a pilot before future assessments with deeper access, but then OpenAI should present a plan for how and when they intend to meet their commitment to employee-like access. And either way, OpenAI is presumably able to go further here.










































