My new investigation maps nearly $3 BILLION in historical and projected funding behind the AI existential-risk ecosystem.

Okay, that should be my new cover image, lol

This 2009 LessWrong meme of Yudkowsky as the ascending Christ was created (or first posted) by user CannibalSmith. Roko responded: “My only question is: who gets to be Judas?” greaterwrong.com/posts/9jF4z…

A must-read on "model welfare": The Effective Altruism mindset, mainly inside Anthropic, that views AI models as deserving of rights and welfare considerations. A lot of interesting quotes here! E.g., "In 2023, OpenAI board member Paul Christiano said the AI industry could be creating a crazy slave trade in which humans profit off the forced labor of digital minds."👀

🔥Burn down the AI labs🔥 The podcaster of "For Humanity," John Sherman, who previously asked StopAI guys, "Why are the executives of OpenAI not charged with attempted murder of ALL of us?" suggests in this new episode with PauseAI to "walk to the labs across the country and burn them down." "Like, literally. That is the proper reaction." "I'm not talking about killing people. I'm talking about destroying the hardware infrastructure." Later on, the interviewee, PauseAI founder Joep Meindertsma, pushes back against the possibility of being associated with illegal/violent/terrorist acts. It will hurt his movement. The podcaster says the message should be, "We are the last humans. Thank you, Sam Altman. You f*cking did it. We are the last humans." Acting upon his idea [from 2:20—"burn them down" 🧨] is left to the podcast's 74k YouTube subscribers. -------------------------------- On March 6, PauseAI will brief some members of Congress. The team will include this podcast host. According to Joep, PauseAI will focus its lobbying efforts on "The AI chip supply chain." The people in Congress (and you) should listen to this exchange first… --------------------------------

The "AI Impacts" survey is back, and its main problems remain. (1). Vague Phrasing. Respondents are still being asked to assign a probability to "human extinction or similarly permanent and severe disempowerment of the human species." That wording bundles together two very different outcomes. On the one hand, there's "human extinction." On the other hand, what exactly can be counted as "disempowerment" of humans?? Different respondents can reasonably interpret "disempowerment of the human species" in very different ways. Yet their answers are collapsed into a number that is repeatedly presented as an "extinction" probability. Problematically, media coverage will be about the extinction part of the question, not the far more ambiguous category of "disempowerment" (and whatever that even means…). One version asked to estimate the probability of this vaguely defined outcome occurring "within the next 100 years." In other words, to reason about an extraordinary, underspecified, century-scale outcome. AI Impacts itself has acknowledged some of these limitations. In a 2024 interview, Katja Grace admitted that regarding the extinction risk question, "We did not make sure that this is an informed estimate," that the participants "are very unlikely to be experts in forecasting AI," and that "There have been quite notable framing effects." The new paper also says expert forecasting ability is questionable: "Expertise… is a poor predictor of forecasting accuracy," and concludes that "it is unclear how valuable their predictions are for accurately forecasting outcomes." (2). Another methodological issue to consider is the sample size. AI Impacts "sent emails to 19,874 addresses and received 2,052 complete or partial responses for a response rate of 10%." The team then kept 1,580 respondents for the main analysis, with 1,502 reaching the final question (78 dropping out earlier). N=1,580 is 7.95% of the emails contacted. Furthermore, no individual "extinction/ disempowerment" question was answered by the full sample: randomized subsets of 744, 392, and 353 respondents answered the three variants. (3). Ideological and Institutional Context. AI Impacts operates within Berkeley's MIRI (Machine Intelligence Research Institute). Eliezer Yudkowsky co-founded this organization and has publicly advocated "Shut It All Down." Katja Grace told The New Yorker that her p(doom) was somewhere “between ten and ninety percent.” It is a highly relevant context. MIRI is explicit about its political objectives. Its main communication goal is to reach policymakers: "the people in position to enact the sweeping regulation and policy we want." Its ambition is hardly subtle: "We don't want to shift the Overton window, we want to shatter it." Another co-author is David Krueger, who assigns a 75% probability of doom and believes that "If we don't stop AI in the next couple of years, it's more likely than not that we will lose control and humanity will go extinct." In the "acknowledgments" segment, the team thanked "Coefficient Giving, Jaan Tallinn, Future of Life Institute (FLI), and others for funding this project." AI Impacts was also previously funded by Nick Bostrom's FHI (Future of Humanity Institute), EA Funds, the Centre for Effective Altruism, and Sam Bankman-Fried's FTX Future Fund. This funding ecosystem is relevant too.

A reminder about AI Doomerism 🧵 (1). Its weak foundation and the unconvincing "doom bible" (2). The deliberate panic messaging and the broader mythology (3). Rationality/Effective Altruism origins and the media's blind spot