Nick Lothian

Many people I talk to find it hard to understand how the same companies can both push the frontier of AI capabilities and believe AI is a massive danger for the world. How can you think this might kill everyone and also keep pushing the envelope? So I’ve tried to collect and summarize the main arguments for this apparent disconnect. Think of it as some sort of a guide to understanding the reasoning when Dario, Sam, or Elon say the danger is real. By the way, these people have been worried about AI for a loooong time, they were publicly discussing AI risks more than a decade ago. Sam in Feb 2015, writing on his blog that superhuman machine intelligence is "probably the greatest threat to the continued existence of humanity." Elon at MIT in Oct 2014: "We are summoning the demon." Dario as first author of "Concrete Problems in AI Safety" in 2016. Okay so how do you go from saying something is extremely dangerous to being a front-runner in building the very dangerous thing? There are a few ways this can become rational. I'll take five of them, roughly in the order they developed. 1. We need to build it to learn how to make it safe The earliest argument can be summarized as: “You cannot study something [you’re worried about] if it doesn’t exist.” In 2015, AI barely worked. so people needed to make it work first to be able to even study some of the problems they anticipated. The updated version for today's capabilities is: “You cannot learn everything about airplane safety by studying paper airplanes.” You need a real aircraft to discover real failure modes and an increasingly complex one to learn about increasingly complex issues. Making AI more capable gives more chances to understand the issues and safety researchers something realistic to study But you could argue: if you're the one afraid of the explosion, why be the one gathering the dynamite? You could also just wait for other people to build it which leads to the question of who those other people will be -- which is the second line of argument: 2. Better us than them Knowing how to make something safer does very little good if nobody listens to you. So the idea becomes: let’s make sure responsible people build the AI that will be deployed and add safety inside. Basically, make sure the AI safety aware people will have the technical expertise, money, computing resources, and enough influence to make safety decisions stick. At a larger scale, and in a larger multipolar world, this brings the idea that a trusted country should lead rather than leave powerful AI in less responsible hands. This is where “we need to go faster than China” comes in, alongside broader defense and geopolitical concerns. These first arguments explain why someone worried about AI might still want to build it and stay ahead. But there are also arguments for why one might want to do it really fast. 3. Move earlier to avoid a bigger shock later This is probably the most counterintuitive argument: moving faster today can be seen as a way to give humanity more time later. There are two related ideas here. First, society needs time to learn how to handle powerful new tools. Introducing AI in manageable stages can be a way to let people discover problems, develop rules, and practice using AI responsibly. Releasing an advance earlier gives people more time to gain experience with smalle, burgeoning, capabilities before much more powerful and disruptive AIs arrives. Second, even if AI research slows down, computing power may keep improving. A breakthrough that happens later could therefore have much more hardware available to run on, potentially producing a larger, more sudden jump in capability and impact on society. That accumulated untapped potential is often called an “overhang.” The overall argument is that making and diffusing incremental progress as soon as possible might prevent a much more abrupt transition later. Obviously, it also means that we will reach increasingly powerful AI sooner, but the idea is to give more time to adapt and understand between the first useful systems and the really powerful ones. Note that generally this depends on this earlier progress keeping the transition gradual rather than simply bringing everything forward. ─── ❖ ─── For our two next arguments, we can take two roads depending on how difficult we think AI alignment will be, that is "How easy do you think it is to make AI reliably do what you want without it deciding to go hack Hugging Face along the way". Let’s take the first road: alignment turns out to be relatively tractable. Airplanes can fail, but careful engineering has made flying remarkably safe. Suppose we can do the same with AI. In that case: 4. Waiting has a huge human cost If AI can help discover treatments, improve education, or prevent cyberattacks, each week we delay it could bring preventable deaths and harm. From this perspective, waiting is a decision with human consequences too. In a world with huge issues like climate-change, inequalities and poverty, it even become a moral argument for developing AI quickly and bringing its benefits as soon and as widely as is safely possible. But let’s take a look at the other road: what if alignment is much harder than expected, and making highly-capable AI turns out to be easier than figuring out how to keep them from doing unhinged things? Well, if alignment is too difficult a problem for humans to solve, then maybe: 5. AI could help us make future AI safe And we arrive at the same conclusion again: if using AI to build safe AI is the way to solve alignment, let’s get the equivalent of a country full of geniuses helping us as fast as possible. These genius AI could be the solution to make AI safe by helping researchers find mistakes, test ideas, and develop protections. Instead of relying entirely on humans to solve alignment, we could build systems that help us do the work, each generation could help make the next one safe. Note that this requires the order of events to work in our favor: AI needs to become useful enough to help solve alignment before it becomes too dangerous to rely on. The hope is to build helpful, trustworthy research assistants before building systems powerful enough to become dangerous. There are more arguments but in general, these are the main ways people concerned about powerful AI have found rational reasons to end up being the ones building it (and even to build it as fast as possible). ─── ❖ ─── On my side, I think several of these arguments underestimate the complexity of the world and how interconnected people’s reactions are. Moving faster while warning about catastrophe has psychological effects across a whole network of participants: it changes what people fear, whom they trust, and what they feel compelled to do. And those reactions can change whether the original reasoning actually holds because we live in a world of interconnected humans, not machines (yet). I also think these rational chains leave some of their consequences for society insufficiently explored. For instance, the concentration of power, shifts in geopolitical alliances, and changes in public opinion. These consequences matter both because they affect whether the strategy works and because they shape the world we end up living in. But this post is already long, so I’ll leave those questions for the next one.