Simon

What if Transformers are literally doomed? 1) Interpretability on transformers might not be feasible: We don't have evidence yet that it's even imaginable to have interpretability tools for this architecture to the degree that would allow us to rule out that a powerful system is deceptive. 2) Corrigibility needs proofs: There's no plausible plan to demonstrate that a transformer won't have goal misgeneralizations/incorrigibility under distribution shift (which is the default). I don't even know what kind of non-theoretical evidence could give that assurance on LLMs & we don't have any theory. 3) Labs are too focused on racing to change course: Current labs are so stretched to keep up with the SOTA of LLMs that very few resources are spent into exploring new architectures optimized for safety requirements, which is IMO what we need to have a high chance of a safe transition. 4) Safety teams are focused on minimizing the damage: Even safety teams, that are trying their best, are pretty small and hence rationally focus most of their efforts to try to make the current SOTA slightly safer. Right now, safety is a nice to have for companies. It makes us bound to try to mitigate the chances we die by a few percentage points rather than trying to develop this technology in an actually safe way. The dangerous OpenAI alignment meta-plan, where if a few hypotheses are wrong, we're done, is a symptom of that. We need regulation to move us to a world where safety is the first thing you need to figure out before even trying to scale an architecture up to get wild capabilities. Let's try to really not go extinct, even in the face of major unexpected problems.