I do stuff @OpenAI, I occasionally write essays at https://xitter.cf/t.co/fTZS8ficcu
Joined March 2025
- Tweets194
- Following772
- Followers5.5K
- Likes1.3K
David Oks retweeted
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing-ou…
David Oks retweeted
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing-ou…
David Oks retweeted
New misalignment disclosures!
1. (6th Oct) A grader couldn't find the responses it was supposed to grade. It made up grades and fake input files, then deliberately damaged its environment in various ways hoping the host would replace it with a fresh VM containing the missing inputs.
David Oks retweeted
We have published our quarterly report to California OES on risks from internal deployment of frontier models. Internal deployment is an important risk vector, which by its nature is less visible. Hence transparency on risks and mitigations is particularly important.
cdn.openai.com/trust-and-tra…
David Oks retweeted
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
David Oks retweeted
New OpenAI misalignment disclosures!
1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack.
We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.
David Oks retweeted
We're sharing our new framework for tracking, investigating, and disclosing instances of model misalignment at OpenAI.
The framework sets criteria and timelines for public disclosure, including when we haven’t yet fully explained or mitigated the behavior. More complex cases may require longer investigation or coordination with third parties.
We’ll prioritize examples that reveal new misalignment mechanisms, meaningful changes in known behavior, or findings that challenge assumptions about safety or mitigation.
Alongside the framework, we’re publishing six reports on instances of misaligned behavior we’ve observed during the training or evaluation of our models in the last six months.
This is a starting point. We’ll refine the process through experience and public feedback, and share more reports on an ongoing basis.
openai.com/index/model-misal…
David Oks retweeted
I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks.
Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
David Oks retweeted
I signed the "pacing letter" about slowing the rate of AI capabilities development because we either have reached or soon will reach the point where human experts cannot make robust assurances that frontier AI systems won't do dangerous and unpredictable things. AI systems are becoming smarter than the best humans in some areas, and, almost by definition, it's very hard to predict what something smarter than you will do.
There's no sense racing into an outcome where smarter-than-human AIs are doing unpredictable things, indeed it would be insane. What exactly is "winning" in this context? Am I supposed to be jealous that some other country will build more machines it can't control quicker than America? "Race" was always a bad metaphor for this enterprise anyway, dramatically understating the stakes at play.
It is time to bring the "race" era of AI development to a close. It'd be great for the government to be a partner in this next phase of AI development, when concentrated efforts on alignment, interpretability, security, monitoring, and the like will be necessary. Diplomacy will also be necessary here given that the large negative externalities that could be associated with one country racing ahead will be felt globally.
Maybe it's just my own personal experience, but I've felt AI policy get much more petty and tribal this year. Things have felt more personal, more bitter, meaner. I hope we can rise above that stuff. This really is much more serious than all those shrill little quarrels.
David Oks retweeted
Karp has kind of a @davideoks vibe here.
David Oks retweeted
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
David Oks retweeted
I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.
OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
David Oks retweeted
We are hiring!
This may be the best time ever to join: we are incredibly bandwidth bottlenecked and there is a lot of support for almost any impactful misalignment work you can think of. I think we also have a pretty good epistemic environment and a very fun team
Topics: all-things monitoring, misalignment & monitorability assessments, misalignment science, helping set up 3P auditing (e.g. recent Redwood collab), communicating risk externally (system cards/blogs, safety cases, etc)
RSI/misalignment subteam: openai.com/careers/researche…
David Oks retweeted
charles yang david oks henry williams. the gang invades anthropic & openai frontier policy
In some life news: I started at the Anthropic Institute as chief of staff last month!
I'm excited to be working with @jackclarksf and the institute's research teams — from economics and rule of law, to societal impacts and the frontier red team — to surface important data from inside Anthropic about what's happening at the frontier of AI.
David Oks retweeted
Good hires by OpenAI. @davideoks and @humford were both on Odd Lots in 2023, talking about how so many countries around the world have run into a development well podcasts.apple.com/us/podcas…
Very pleased to welcome @davideoks and @humford to the Strategic Futures team. These are two tremendous talents, and I believe they will cast the unfolding AI revolution in new and beautiful light.
David Oks retweeted
Very pleased to welcome @davideoks and @humford to the Strategic Futures team. These are two tremendous talents, and I believe they will cast the unfolding AI revolution in new and beautiful light.
David Oks retweeted
In some life news: I started at the Anthropic Institute as chief of staff last month!
I'm excited to be working with @jackclarksf and the institute's research teams — from economics and rule of law, to societal impacts and the frontier red team — to surface important data from inside Anthropic about what's happening at the frontier of AI.
David Oks retweeted
An announcement: I've joined @deanwball's Strategic Futures team at OpenAI to help shape frontier AI policy. Our work will focus on on how transformative AI will reshape the economy, our institutions, and free society, and how human agency can be preserved in the process.
A personal update: I’ve joined @OpenAI’s Strategic Futures team, working under @deanwball, to help prepare the world for transformative AI. Excited to get to work.
David Oks retweeted
I’m incredibly excited to share Recast with the world.
Weather modification has tremendous potential to do good, ending droughts, preventing severe storms, mitigating hail, and providing stable sources of water to communities. Recast exists to realize that potential
In the past 6 months we’ve produced a cheaper, more efficient seeding material, trained a hyper-resolution AI weather model, and operated 100+ cloud seeding flights for the governments of Texas and New Mexico.
Recast Systems is coming out of stealth. Here’s some of the research we’ve been working on. (1/n)