Chi Nguyen (@NguyenSquared) profile picture

Redwood Research (@redwood_ai) check out https://teapaw.org/t.co/Fd3TkFXWs5

Joined April 2020
Photo by Chi Nguyen (@NguyenSquared)Photo by Chi Nguyen (@NguyenSquared)Photo by Chi Nguyen (@NguyenSquared)Photo by Chi Nguyen (@NguyenSquared)Photo by Chi Nguyen (@NguyenSquared)Photo by Chi Nguyen (@NguyenSquared)Photo by Chi Nguyen (@NguyenSquared)Photo by Chi Nguyen (@NguyenSquared)
Chi Nguyen retweeted
Over the last year, I’ve become increasingly scared about research sabotage on safety tasks by future AI models. Some personal thoughts: We are currently struggling to fix safety problems that are relatively easy to notice and train away. It will take far more work before any company can reliably catch, and successfully train away, strategic underperformance on safety research. The safety community is dramatically underprepared for this and for the related problems of underelicitation and non-strategic underperformance on safety work. I’m planning to write more on this topic soon. Because of the small sample, underelicitation, and other limitations, it’s difficult to draw very rigorous conclusions from the empirical data of our paper alone. A more skilled human researcher would have produced significantly more covert sabotages than I did; our auditors would also have likely done much better if they had more research experience, more time, or more powerful models. Even so, subjectively, working on this was a big update. I had previously thought that forcing the attacker to log every action, and letting the defender run unlimited experiments, would significantly advantage the defender. Most relevant results to me: the attack selection results suggest that a strategic saboteur could guess fairly accurately which attacks would be easy to detect, and so attack only when it would be very hard to identify. Also, the fact that auditors were much worse at identifying ‘minor’ problems targeted at changing the results, compared to ‘major’ ones, suggests that generic sandbagging might be even harder to detect than the targeted sabotage I studied here.
New paper: It is difficult to detect when AI safety research is sabotaged and even harder to identify the specific sabotage. Even with access to the full research log and the ability to run experiments, human auditors usually couldn’t distinguish benign from sabotaged experiments or determine how the sabotage was done. Frontier AI models likely do better, but still miss most sabotage.
8
10
89
2,265
Chi Nguyen retweeted
METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
175
1,072
595
6,016
4,418,835
How do LLMs reason about playing games against copies of themselves? 🪞We made the first LLM decision theory benchmark to find out. 🧵1/10
2
18
2
102
11,154
Chi Nguyen retweeted
How close are current AI agents to automating AI R&D? Our new ML research engineering benchmark (RE-Bench) addresses this question by directly comparing frontier models such as Claude 3.5 Sonnet and o1-preview with 50+ human experts on 7 challenging research engineering tasks.
14
170
83
827
446,346