Simon Pickup

Two things I’ve been thinking about regarding AI: • People often highlight the risks involved in AI, which are surely very real. I’m struck, however, by how hard it is to understand the contours of the positive scenario. There are easy-to-perceive and frequently invoked elements of a positive vision — more diseases cured, more material abundance, etc. — which sound great, but these are by themselves just ingredients. I’m curious what the larger equilibrium is supposed to look like. Do people work? If so, why? What is their marginal contribution? If not, how are resources allocated? What is scarce? What is the ratio between humans and AIs? Are humans in control, whatever that means? One sometimes hears the Culture novels invoked as one scenario, but I’m not sure they depict a desirable outcome. (AI runs everything and humans compete in games? Iain Banks himself said that the humans in the Culture are “somewhere between passengers, pets and parasites.”) There is Richard Brautigan’s famous poem, but do we want to be “returned to our mammal brothers and sisters”? (Isn’t that rewinding Genesis?) We could say that it was hard to imagine the world of 1900 from the vantage point of 1700, and that’s assuredly true. But nobody in 1700 was trying to recast humanity itself within a couple of years — things necessarily unfolded slowly and by increments, and therefore with the possibility for error correction. Given how rapid the rate of progress in AI may be, it feels strange for the upside case to be so opaque. • There is increasing discussion of whether AIs are conscious or how we would know whether they are, with conclusions there often assumed to be relevant to matters of moral worth. At first blush, this seems a rather arcane, benign, or even absurd issue. (Remember the response to Blake Lemoine’s fears.) Setting aside the terms of the debate itself, or the relevant demarcation criteria, it seems to me a quite high-stakes question, for an affirmative answer — that is, granting moral worth to AIs — would presumably mean that the aggregate worth of AIs would soon outweigh that of humans in any utilitarian calculus. The AIs will be exceptionally wise and intelligent, and vastly more numerous (especially once one considers the imminent future and the AIs not yet born). As such, even epsilon of moral worth will multiply to a lot once one considers the quadrillions that could soon exist. (This is effectively a distributed version of the Nozick’s utility monster.) If this conclusion is unpalatable, I think one of three propositions needs to be rejected: the premise AIs can have moral worth; utilitarianism itself; or the belief that there is even a possibility that they will be far more numerous than humans. With both of these matters (contours of future society; AI moral worth), there is the possibility of dramatic change. Today, people argue about datacenters. I’m curious how popular support for AI will evolve as more people come to understand these medium-term questions and the relevant stakes. What is the vision for AI that most people would be excited about?