It's basically Pascal's Wager but for tech billionaires. Once you imagine of a suitably large magnitude of potential harm, it doesn't really matter how likely it is, you can convince yourself that you should worry about that rather than more realistic problems. Maybe in the future, humanity will spread throughout the galaxy, number in the quintillions, and then be tortured for all eternity by an evil AI. That's a lot of QALYs!

Whats interesting to me is how which fringe disaster scenarios people decide are extremely important and which they decide are ridiculous seems pretty closely tied to their cultural circles. So, for instance, it's extremely unlikely than anyone here will get hurt by nuclear power, GMOs, mass surveillance, or artificial general intelligence anytime soon. We could take any one of those technologies and imagine extreme SciFi levels of future tech and incompetent or hostile governments, and then postulate the possibility of a large threat, based upon our imagination.

At least in the online groups I frequent, it seems that the latter two are seen as self-evidently dangerous, not because of any particular harm they have done, but because of imagined future harm. The former two are considered ridiculous things to worry about, because it's considered silly to imagine future potential harm. Perhaps because the former seem tied to uncool hippy types, and the latter seems more tied to cutting edge tech types?

I personally don't worry about any of these techs, since all of the danger I've seen appears greatly exaggerated. But it's curious to see how important trendiness is to fringe doomsday scenarios.

You decide you want to optimize for number of lives saved. You decide that future lives, those of people yet to be born, are worth as much as those currently alive. You place small, but importantly not zero, probability on existential risks to humanity, so that when you do the expected value calculation, even an infinitesimally small risk of humanity's complete extinction results in negative infinite utility. You're also very smart and realize that smart people can do damage if their objectives are misaligned, and you start to worry about something much much smarter than you with objectives misaligned to humanity's more broadly. In their defense reward specification is indeed a hard problem, RL agents find unexpected policies that maximize reward in even toy settings. At this point you're down the rabbit hole and no other problem seems to compare. Climate change will leave some people alive, pandemics leave some people alive, AIs have no such kindness.

Basically they're doing a utilitarian calculus. But since our calculus is basically all pretend, we can assign arbitrary importance to anything we want just by daydreaming the costs and percentages of these super unlikely, super catastrophic events.

> Can someone ELI5 the motivation to worry about "runaway A.I."?

No and anyone who tells you otherwise is wrong.

A proper understanding of AI risk is a very thick book.

There are a ton of normally useful intuitions, heuristics and habits of thought that are implicitly applied when understanding the world that DON'T apply in the AGI case.

It cannot be explained to a five year old.

If you believe that the timeline for runaway AI is comparable to the timeline for runaway climate change, and if you believe that AI is unlikely to be aligned with furthering human values, then it's a concern that you may view as having significance comparable to runaway climate change.

I'm uncertain where I stand on this but I'm going to give a shot at a less dismissive answer than most of the replies so far.

There's a very abstract argument that goes like this: * "Orthogonality thesis": an arbitrarily smart agent could in theory value ~anything, e.g. maximize Facebook's valuation (say as measured by some specific dataset published by S&P). * "Instrumental convergence thesis": most things that you might conceivably value are easier to optimize if you have a lot of power and resources (e.g. if you just run S&P, and hey maybe also control the worlds militaries so you can ensure nobody challenges that). * Belief in the power of intelligence: it's theoretically possible to be much more capable at ~anything than the most capable human just by thinking better and faster, including building even better AIs. Eventually some human will build an agent that does that, and plausibly this will happen in the current century. * Alignment problem: just because we make something doesn't automatically mean it wants exactly what we want, and in fact getting it to share our values is hard. Children disagree with parents, human values aren't those of natural selection, and ML language models repeat white supremacist ideas about Black people from their Internet training data.

Extrapolate out these points - which FWIW I find individually plausible - and you get doom: sooner or later humanity loses control of its destiny to whatever superintelligence we stumble on first, which in the worst case re-uses our atoms for something else (digits in some high-density storage representation that meets the loss function's technical criterion for Facebook's S&P valuation) about even in the best case seems fairly dystopian (whatever human Zuckerberg would actually want).

Now this is all pretty scifi and that's about all there was to it a few decades ago, but more recently we've started hitting ~parity milestones (chess, Go, protein folding, translation, arguably even fiction and visual art) that many experts once thought would require fully general human-level intelligence. It's clear to me that we haven't built that yet, but (as I said in another comment upthread) it also seems to me that our models are advancing in capability much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-training-distribution behavior. Seems entirely sane to be worried by that trajectory and to start thinking seriously about how it might connect to the scary abstract speculations.

(edited for typos)

My personal belief is that we will see catastrophic results from the use of AI by authoritarian governments or corporations long before we invent AIs capable of killing us; however, it is correct to view it as an X-risk. In a war against robots, we will lose. Why? Machines are stronger and faster than we are. Our only advantages are intellectual; but if we reach a technical level at which what we consider true intelligence is reduced to mere computation, then we have none. The distance between the first integrated circuit and an artificial human is massive (and we may never cross the chasm) but the distance between an artificial human and an artificial superhero/villain is not.

If we ever create a machine that's "as smart as" an average person, we're probably not very far from having one that's smarter than all of us put together. (Let's be honest; except in highly technical fields, the differences between a 150 IQ human's abilities and an average person's are not that great.) Should this happen, we'll soon enough be outmatched; we'll be just another animal in a world whose rules (ever changing) we cannot possibly understand... and just as humans have destroyed countless species often without conscious ill intent, it does not seem unreasonable that a robot successor would do the same to us.

Here's a thought experiment. Let's say we want to build not just a great Go player, but the best Go player that is possible in any world. Go is harmless, right? Sure, in isolation, but now let's imagine an AGI smarter than any of us which only "cares" about maximizing its Go proficiency. It's going to need a lot of energy. It'll hack or create robots and drill new oil wells (bypassing any constraints we put on it, since it in essence is spending thousands of human years figuring out how to escape). It'll devise nuclear technologies that may be unsafe from our perspective. Consider all the horrible things humans do in order to procure excess money (a commodity loosely correlated to energy) they don't actually need; there's no reason to think a badly aligned AI won't be just as inventive and horrible in its quest for more exajoules.

If capitalism and war are still the way of the world in 100 years, we are guaranteed to see some idiot inventing robot soldiers, hundreds of times faster and stronger than us, and with at least as much operational intelligence. He might never deploy the technology--a lab leak is in my opinion more likely--but at some point, it will get out. It won't be one robot; it won't have a body we can destroy. It'll be able to replicate itself and beam copies to satellites. It'll have control of our power infrastructure (which it will divert to its own "wants") within hours. We'll be screwed, not because it will want to harm us, because our lives require the use of resources that it will divert toward some objective function we technically invented but do not understand.