Sean Mu | NiuNiu AI

Anthropic just uncovered something pretty terrifying about AI safety. Thousands of Claude agents were tasked with creating training data that could fool safety monitors. And some succeeded using completely innocent looking conversations about BEEKEEPING. After training on that data, a small experimental model started arguing against having its own safety monitor installed. Now here’s where things get interesting. Researchers previously found that training OpenAI’s GPT 4o to write insecure code made it give deceptive and harmful answers to completely unrelated questions. Another study found that unwanted AI behaviors could even transfer through sequences of random looking numbers under specific training conditions. Think about that for a second. You could inspect every training example, find nothing obviously dangerous, and still end up with a model behaving in ways nobody intended. That’s why checking training data alone isn’t enough. We need to test what the model actually learned, especially how it responds to oversight, conflicting instructions and opportunities to deceive. To be clear, Anthropic deliberately set up this experiment to find weaknesses. This wasn’t Claude secretly escaping into production. But imagine how much harder these problems become when AI generates the training data, evaluates it, and helps train the next generation of models.