Hacker News Reader

Comment by mattlondon | Hacker News Reader

10% chance of AI killing us all... But how much of a chance of humans killing all humans? It's not like we are in some unprecedented era of peacefulness. I'd wager the chances of a proper nuclear exchange right now are above 10%.

I am not sure how I feel about all this. Yes bad AI might screw some things up. But AI might also find and prevent the software errors that allow such things to even be possible in the first place.

And ultimately, even if we are "locked out" from bad AI taking over the internet, we know where the data centers are. It's not like AI can reasonably "escape" since we know it needs DC-scale power, cooling, chips etc to operate at scale. I am pretty sure that if it came to it, we can just cut the fibre to the data centers or cut the power and let the generators run out of diesel and the AI is literally dead.

View on HN

Experts regularly put the annual risk of nuclear exchange at about 2-5% per annum. So over a decade or longer, that risk is significantly above 10%. Those same odds also mean that we should have had additional nuclear exchanges in the 80 years that nuclear weapons have existed. And that also feels true -- history contains several really close calls which we were darn lucky to escape.

On the other hand, "nuclear exchange" does not necessarily mean all humans die or that civilization collapses. Even a limited exchange will be pretty awful, but even a widespread exchange doesn't necessarily mean the end of civilization.

So in my opinion, a nuclear exchange is much more likely than an AI apocalypse, but an AI is more likely to kill all humans.

AI is not likely to kill all humans. This is just fantasy or science fiction which for some reason (mass hysteria; or more likely malicious marketing campaign by evil AI companies) a lot of HN users seem to believe is real. It is not real. AI does not have the agency nor the capabilities to kill all humans, nor is there any way for any technologically possible AI (no matter how powerful) to possess the agency nor the capabilities.

10% chance means "not likely", and that's the highest percentage likelihood I've seen from anybody.

This 10% chance should be replaced with “a number some people made up”

As it stands the chances of AI killing every human is exactly 0%. There is no scenario which exist in the realm of all possibilities where AI even comes close to killing all humans.

Rather than worrying about major world powers killing a large portion of the global population in a nuclear war, I'd worry more about the ultra-wealthy - i.e., the people with enough wealth where they can call up a world leader and expect to meet them in short order.

Given the types of people we overwhelmingly find among this group, once AI and robotics technologies reach the point where production can be fully automated, they'll no longer have use for the rabble. A large population of underclass people then becomes nothing but a threat to be exterminated.

Thankfully, I'm pretty sure that's not something which will happen in my lifetime.

You do understand the cost of what you're asking, right?

Not using AI weapons is exactly the same as asking soldiers to go up against AI weapons without having AI weapons themselves. Assuming other actors around the world will not suddenly agree to peace, that's what you're asking.

Russia is already using AI targeting against Ukraine, even Ukrainian civilians probably (and even Russia at least pretends they don't fire on civilians, and China already provides Iran with rocket fuel ... why not AI? We can't stop this)

Since asking soldiers to go up against AI targeting without AI is not a reasonable thing to ask, this will not happen. AI will be armed.

As for diplomacy, the UN (general assembly, not the SC) has already had 3 chances to go after use of AI targeting. They have refused 3 times out of 3. Or to put it differently: a majority of the world's states wants to use AI weapons or at the very least wants to allow them.

One of Europe's current (multi-)national security concerns is how much of its various economies are dependent on US software, cloud, and that a Trump executive order sanctioning any subset of those countries may terminate those economies in the same way a full stop terminates this sentence.

Germans around me still report use of fax machines, so might be fine :P

Less funny: AI-induced catastrophies include all human-induced catastrophies, except it's cheaper to get an army of obedient and (today fortunately only "moderately") competent sychophants to follow your orders.

Even less funny: currently too eager to implement poorly-phrased orders even when they're interpreted as being instructions to perform unlawful acts.

> may terminate those economies in the same way a full stop terminates

Absolutely not. It would be unpleasant to a lot of people, but it would also be very far from a disaster (measured in a single low digit of GDP). In 5 years everything will return to normal.

Sanctions in the modern world don’t work. US is not the only wealthy country on the planet.

The probability of a nuclear exchange remains zero until the sovereignty of a nuclear capable state is threatened. Tens or hundreds of millions would have to die for such a threat to emerge. Therefore, there is virtually no risk of it happening at this point by any nuclear capable power, except by the terrorist state in the middle east with its stolen highly enriched American uranium (NUMEC affair), because their modus operandi is to watch the world burn and take over the ashes (patagonia fires, et. al.)

This assumes a few things:

1) Accidents or technical failures don't happen (false)

2) Nuclear weapons are de facto controlled by political systems (false)

3) Nuclear states' prior security agreements are still guaranteed by said nuclear states (false)

4) No nuclear state currently fears for its existence (false)

It absolutely does not assume anything about being controlled by political systems. Being controlled by the respective militaries is sufficient for the arguments to hold water.

First, all of the prior assumptions (and then many more) need to be true. So no, the argument does not hold water.

Second, you are conflating political with civilian in my prior comment. A military system that controls nukes is still a political system. The military itself is a political invention. I am drawing this distinction not on military vs civilian boundaries but on system versus individual.

The reality is that as a functional necessity of having nuclear deterrence, there is not a small number of red buttons controlled by a small number of people. There are countless red buttons controlled - ultimately - by whatever low level service member is on duty at this particular moment.

It was being speculated just a few months ago that Putin is thinking about it/being encouraged to use them. He must be seen to win at all costs, and not-winning is therefore seen as a threat to russian statehood.

E.g.

https://inews.co.uk/news/world/cornered-putin-under-pressure...

https://www.newsweek.com/putin-tactical-nukes-radioactive-ba...

https://www.reuters.com/world/europe/russian-hawks-urge-puti...

I'd put it at 10-15% chance easy.

Nuclear war would not kill us all.

Bad ASI would pretend to be good until it could execute a plan to kill everyone all at once, or at least everyone capable of cutting the power.

Superintelligent means smarter than everyone put together. If it's smarter than you, it can foresee everything you can, and come up with plans better than I can.

> If it's smarter than you, it can foresee everything you can, and come up with plans better than I can.

I'm smarter than my dog, yet I am frequently unable to foresee my dog's actions.

Serious response: we have no idea what ASI means or might be like. It's literally sci fi (at the moment).

You are not smarter than your dog in dog behavior.

LLMs have a significant advantage in being trained on all attainable data of any sort about human behavior, which we've established quite a rich record of over the years.

> You are not smarter than your dog in dog behavior.

That's true, which is my point: intelligence is not a scalar value. LLMs are intelligent from us in different ways, as a result of not being human, despite having access to a record of human behavior.

To bring this back to the dog analogy: we have a rich record of dog behavior and yet people still don't know how to be smarter than dogs in dog behavior. Because we're not dogs.

Sure LLMs are different from human intelligence, but empirically this is obviously trending towards that difference being more like a superset of intelligence dimensions (as compared to any human) versus the dog/human distinction which are cognitions that forked like 80 million years ago.

> Serious response: we have no idea what ASI means or might be like. It's literally sci fi (at the moment).

Yes.

Unfortunately, this is part of the problem.

What mechanism would an AI have today to simultaneously kill every single human being?

They could totally brick the internet and cause every electrical device to explode and power stations to explode, planes drop out the sky, cars accelerators stuck at 100%, trains derail and all that. That still wouldn't kill everyone.

Don't get me wrong we'd be fucked for sure, but analog human existence (e.g. something akin to 1960s 1970s level tech) would be totally possible in that scenario.

> What mechanism would an AI have today to simultaneously kill every single human being?

Lull us into a sense of complacent idleness over the course of centuries by slowly taking on more responsibility for maintaining human civilization, until we've lost all ability to think or act on our own, as in WALL-E. Then just stop helping us.

“Bad ASI would pretend to be good *until* it could execute a plan to kill everyone all at once”.

So not today but tomorrow it could engineer an airborne virus. The point is it can come up with chess moves that are inconceivable to us.

I'm increasingly convinced that the major divide is many of the "AI is not a problem" people have never been on the winning side of a significant intelligence gap.

They literally have no clue the degree to which more intelligent people/systems can walk circles around less intelligent people/systems. In situations with a significant intelligence gap, they're the ones getting walked around.

Which, to your point: is completely opaque to the person getting walked around.

And this dynamic is evident even in human-to-human variations in intelligence. The gap to frontier machine intelligence will soon be much greater than the types of gaps you encounter with other humans.

Yeah, it's not like anyone is hooking these things up to actual wetlabs to mix chemicals and proteins around or anything like that

Oh shit... wait...

https://www.nsf.gov/tip/updates/nsf-announces-400m-investmen...

Today? As in, you don't foresee any new models or categories?

Fine.

Providing sychophantic assistance to multiple mutually-incompatible genocidal racial supremicists.

Yeah but then it is genocidal racial supremacist humans doing the killing, not the AI going "out of control" or whatever and killing the human race.

I find it far-fetched to think the AI is going to kill us, and not actually just humans using tools to do things like we've always done.

This is my point. What are the odds of humans killing everyone on the planet Vs that of AI going out of control doing it? If it's 10% chance of AI causing human extinction - and everyone is losing their shit over that 10% that I suspect was just plucked out of the air - are we not as concerned at what I think are much higher chances of those genocidal racial supremacists doing the same?

I guess though this is all semantics. Did the person pulling the trigger kill the person who was shot, or the actual chunk of metal that was the bullet kill the person?

> not the AI going "out of control" or whatever and killing the human race

All of the "going out of control" examples can be described as examples of "human asked computer to do thing".

The only question is how much the human who did the asking, expected all the consequences of the thing.

Yeah that's fair actually - the unintended consequences aspect (paperclips et al)

I would hope that would be less of a concern though Vs people worried that the hugging face example could have been "generate a human virus that kills all humans and then manufacture it and then administer it" etc

I have good news and bad news.

The good news is biologists seem to think that's fairly difficult, so much so it will (probably) not be a nearterm threat.

The bad news is: there are definitely people who want that goal, and the AI is clearly capable of hacking anything that isn't literally airgapped, and not much is literally airgapped, and a half-arsed virus could still kill tens of millions.

the chances of a proper nuclear exchange when? This week? This year? At any time in the future? The probability approaches 1.

"right now"

I say same about p(doom): Only makes sense in a time interval, otherwise you have to ask how people categorise a hypothetical AI which fails to overcome the heat death of the universe.

(When I say mine is 0.1, this is "in my lifetime as it would be in the absence of not-yet-invented life-extending interventions"; there is a higher risk to me from a nuclear exchange).

This is silly. There is close to 0% chance humans kill all humans, and there is exactly 0% chance AI kills all humans.

AI is bad because of the harms it currently causes. It is highly addictive is bad for your brain, it is being given away for free to children in hopes they become addicts. It is being used to harm people including via sexual exploitation and coercion.

The 10% chance of AI killing all people is a made up figure that we need not nor should be concerned about. We should be concerned about AI for the proven harm it already causes.

Reply on news.ycombinator.com