
Redwood Research (@redwood_ai) check out https://teapaw.org/t.co/Fd3TkFXWs5
Joined April 2020
- Tweets32
- Following71
- Followers90
- Likes1.3K
Chi Nguyen retweeted
Over the last year, I’ve become increasingly scared about research sabotage on safety tasks by future AI models. Some personal thoughts:
We are currently struggling to fix safety problems that are relatively easy to notice and train away. It will take far more work before any company can reliably catch, and successfully train away, strategic underperformance on safety research.
The safety community is dramatically underprepared for this and for the related problems of underelicitation and non-strategic underperformance on safety work. I’m planning to write more on this topic soon.
Because of the small sample, underelicitation, and other limitations, it’s difficult to draw very rigorous conclusions from the empirical data of our paper alone. A more skilled human researcher would have produced significantly more covert sabotages than I did; our auditors would also have likely done much better if they had more research experience, more time, or more powerful models.
Even so, subjectively, working on this was a big update. I had previously thought that forcing the attacker to log every action, and letting the defender run unlimited experiments, would significantly advantage the defender.
Most relevant results to me: the attack selection results suggest that a strategic saboteur could guess fairly accurately which attacks would be easy to detect, and so attack only when it would be very hard to identify. Also, the fact that auditors were much worse at identifying ‘minor’ problems targeted at changing the results, compared to ‘major’ ones, suggests that generic sandbagging might be even harder to detect than the targeted sabotage I studied here.
New paper: It is difficult to detect when AI safety research is sabotaged and even harder to identify the specific sabotage.
Even with access to the full research log and the ability to run experiments, human auditors usually couldn’t distinguish benign from sabotaged experiments or determine how the sabotage was done. Frontier AI models likely do better, but still miss most sabotage.
Chi Nguyen retweeted
METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
Chi Nguyen retweeted
How do LLMs reason about playing games against copies of themselves? 🪞We made the first LLM decision theory benchmark to find out. 🧵1/10







