Slightly half-baked analysis: When thinking about whether misaligned AI poses an existential risk, it’s important to distinguish three hypotheses: 1. Aligning AI systems is easy. 2. We will build catastrophically misaligned AI systems with the power to wipe us out. 3. Aligning AI systems is a hard but tractable iterative engineering problem. People sometimes point to examples of costly misalignment like Hugging Face as evidence for (2). But although they're more likely under (2) than (1), they're not obviously more likely under (2) than (3). And (3) arguably does a better job explaining the full body of evidence: there are relatively few examples of harmful misaligned AI behaviour, even though millions of people use frontier or near-frontier models every day, increasingly for complex agentic tasks. Combine this with the fact that (3) should plausibly have a much higher prior than (2). It anchors on expectations from previous technologies (the normal technology perspective), whereas (2) asks us to believe something really extraordinary: that we’ll accidentally build systems with motives catastrophically misaligned with ours, plus the power to conceal and then act on them. Put together, then, you get a simple Bayesian argument for a low p(doom) given the current evidence. On likelihoods: advocates of doom can argue that the low base rate of costly misaligned behaviour is exactly what the doom argument predicts. Maybe AI systems won’t act seriously misaligned until it’s too late. But then it seems inconsistent to point to current misalignment as evidence for doom. On priors: different theoretical frameworks would obviously lead to different expectations. E.g., someone who assigns high probability to (2) might ask: wouldn’t it be surprising if a less intelligent species could control a more intelligent one? However, to me, this assumes what is at issue: it would be extremely surprising if we accidentally built AI systems that stand in the same relation to us as we stand to less intelligent species. Maybe it will happen, but extraordinary claims like this require extraordinary evidence; they shouldn’t shape what one expects before examining any of the evidence. Alternatively, one might argue that confidence in (2) is rooted in highly theoretical arguments (e.g., about instrumental convergence, the inherent nature of advanced rational agents, etc.), not specific patterns of available evidence. Fair enough. But then what, precisely, do these arguments predict about what we will observe in the future? If no evidence could ever be diagnostic (until it's too late), it's plausible that we should have low confidence in the entire methodology. What does this analysis miss?

My final question: What would have to happen over the next five years for Steve to conclude that Scott Alexander and others who share his views were right and he was wrong?