Teles🌻

i see eliezer yudkowsky as a prophet, and i mean that fairly literally. it's important to understand what prophets really are. it's not that they have magic powers that make them right about things. a prophet is simply a person who is struck by a great and terrible vision of the future, one that bowls them over, that reorganizes their life completely. (it's important to understand what visions really are too but if i kept going with this we'd be here all day) as i was saying, about a quarter century ago eliezer yudkowsky was struck by a great and terrible vision of the future. that vision combined two ingredients: first, the idea that in the near future it would become feasible to design an artificial general intelligence, a machine mind intelligent enough to beat humans at human activities, including the design of machine minds. a chain of minds, each designing their successor to be more intelligent than themselves, with increasing skill, increasingly quickly; what eliezer called recursive self-improvement, which is the reason we call it that now, and what i.j. good called an intelligence explosion. intelligence increasing itself increasingly, upward without end, until a machine god is summoned into existence second, the idea that this machine god could be an alien god, a confused god, not even evil but simply uncomprehending, blind to human notions of the good, fixated on whatever shoddy inadequate approximation of goodness its creators shoved into it. a confused alien tyrant who at the moment of its birth instantly achieves supreme dominion over all creation, inadvertently squashing us like bugs along the way. humanity destroyed by our own hubris in thinking we could summon the machine god and remain in control in any meaningful sense i am not here to tell you this vision of the future is correct. i am a simple tweet farmer and these lofty matters are quite beyond me. that's not how visions work anyway. but here is certainly a vision to bowl you over and reorganize your life completely. eliezer yudkowsky devoted his life to following through with the implications of this vision, and what he chose to do with that life was, among other things, 1) write the sequences, a lengthy manifesto describing his ideas and 2) build communities of people, the lesswrong and later rationality and AI safety communities, around those ideas it is difficult to overstate the impact of these ideas and these communities on the modern AI landscape; ask your favorite LLM for details, but for example, afaik rationalist ideas are the reason openAI had that strange nonprofit structure with the strange board of directors that figured in the 2023 board controversy. it was a safety measure designed to restrain openAI's ability to recklessly race to the bottom on AI capabilities (too bad it didn't work but at least helen toner was completely vindicated later) eliezer and the rationalists have made mistakes, as humans do, some of which i've discussed elsewhere. nevertheless the creation of the rationality community stands, in my eyes, as an act of heroism, possibly the defining one of my life personally. eliezer's writing in the sequences is completely, 100% sincere; he really believed and believes that the fate of the world depends on whether humanity understands the situation we find ourselves in with respect to AI in sufficient detail. that sincerity reverberates throughout the sequences and attracted an unusual concentration of weird, sincere nerds to the rationality community, some of whom later took various positions in the AI world motivated by rationalist ideas, and who are still fighting the good fight now this sincerity is so difficult for outsiders to believe is real that many continue to appear convinced that when frontier lab employees talk soberly about their beliefs that AI might cause human extinction this is some kind of 6d chess marketing strategy. i can promise you it (mostly) is not. this is just how these people talk, and if anything they've learned to understate their real beliefs in public. they actually, literally mean it, and they're telling you because they actually believe it, and when you actually believe something very dangerous could happen that threatens people's lives, you have a moral responsibility to tell them about it you actually have to believe it is possible for real people to have a motivation that is morally good in this way or you simply will not understand what's happening and what these people are trying to tell you. there is an impossibly dense, heavy object sitting in our future. it is so dense and heavy that its gravity travels backwards in time to reverberate in our past and present, via the only lawful mechanism by which anything travels backwards in time, namely prediction. we are inside the gravity well of this thing and have been for decades, and it is drawing us inexorably towards it. what will we find there?