Time from conception to testable super soldier candidate is how many years? At what year did this first become possible and how many of those decade plus long testing cycle itterations would be required to develop an army of good clones and fit them into that all white body armor? "Well enforced" is holding a LOT of weight there. History of underground nuclear weapons programs shows that nations do not actually abide by tech limitation treaties. (This includes some of the current nuclear powers who made it over the line by cheating.) The idea that you can bomb suspected facilities only deters those that don't have their own deterrent. Nuclear detterents have been seen to work. So that is 9 independant labs in nations with nukes right there, even if you don't accept the idea that non governmental secret labs will also exist (they will) and that some nations without nuclear capabilities will try anyway because they missed the boat on being nuclear powers and want to join the big boys. So lets just say 9 labs and look at what p(doom) per lab we exceed a 90% chance of having everyone die because you put your faith in "well enforced" international treaties and one of the nuclear powers who will still feel free safe in continuing research gets it wrong. For N independent trials, each with failure probability P, the probability of at least one faillure is: 1 - (1 - P)^N Set this equal to 0.9 with N = 9 1 - (1 - P)^9 = 0.9 (1 - P)^9 = 0.1 1 - P = 0.1^(1/9) P = 1 - 0.1^(1/9) P ≈ 0.226 So if your per secret lab p(doom) exceeds 22.5% you are now at p(doom) 90% and have likely committed probabalostic suicide. What is your p(doom) now? Current conditions have some level of alignment research sharing going on. That is where you get your scary AI swarm stories from. So right now, collectively, we are in a better situation than secret labs with no sharing from other bright minds working on the same problem. Take your current p(doom) and increase by the amount that makes sense to you for a singke rogue secret lab's p(doom). Are you over 22.5% ? I did not invent probability mathematics to support my ideology. My reasoned beliefs in this matter come from historical observations of government actions in similar arms races and my mathematical knowledge. Yours come from ideological submission to an authority who continues to fail to address the above argument, either because he knows he already fucked up and probably doomed us the moment he was chatting with Bernie Sanders, or because he doesn't really belive the things he says and just says scary things to get the attention he craves. The math doesn't lie. Sure, the history on limitation treaties might not repeat itself. It is possible. But is it probable? That history is certainly the only information we have to go on, other than lack of supersoldiers, which I have pointed out is not as quickly or easily developable a tech as nukes or AI. Are you willing to risk going from p(doom) 22.5% to p(doom) 90% betting that history won't repeat itself? A "pause" is probablistic suicide. The rational part of your brain that followed this argument already knows it, while the tribal instinct part is trying to send your thoughts in any other direction, because accepting heresy means losing group, and on evolutionary time scales, losing group meant almost certain death. So right now, your evolved mind is quite litteraly reacting to my reasoned logical argument as if I am someone who wants to kill you.

"I came in skeptical and left persuaded that the core arguments survive serious scrutiny. The renormalization framework alone is worth the read." -ANTHROPIC CLAUDE 4.6

This is long but really good. Everyone interested in "AI alignment" and "Superintelligence" should read this. buttondown.com/apperceptive/…

My first reponse to Roko's basilisk was "This is such a stupid thought experiment that I must now demand that future artificial superintelligence simulates refunding me the time I spent reading about it." The other issue with infinity thumbs (infinitimidation) is that they always somehow then require a specific entirely unsupported "solution" (e.g. the human sacrifices to Quetzalcoatl you mention). We describe this as always being employed together with the classic "political syllogism" (undistributed middle fallacy): Something must be done! This is something. Therefore, we must do this. So that any insane course of action can be imposed in the wake of the inspired fear. Y&S use this to get from "any mistake building ASI ends us all" to "we must bring government in to pause this yech" even though the obvious (based on all historical observations) result of government control of this tech would be dozens of secret/classified labs around the world rather than open work that might combine our efforts into a single open roll of the dice rather than many hidden ones. Even if (especially if) you accept their scare story, their solution, given even a few seconds of rational thought, not paralyzed by fear, turns out to be probabilistic suicide.

We call it an "infinity thumb" placed on the scales weighing an argument's merits, discover two more infinity thumbs on the pro-ASI side to outweigh Yudkowsky and Soares' one, but then point out that two infinities don't really outweigh one and all must be renormalized.