Hacker News Reader

Comment by Thorentis | Hacker News Reader

Just wait until somebody think it's a good idea to declare war, based on the advice of an AI. That scares me more than anything.

Or launching nuclear weapons based on the advice of AI (or allowing AI to control them altogether). All out nuclear war has so far been avoided multiple times, because of an actual human making a split second decision at the right moment.

View on HN

Human civilization will die an embarrassing death. Maybe we will even earn the equivalent of a Darwin Award on Gwelkklak 10, "Died by Own Avoidable Science Fiction Trope".

Maybe the answer to the Fermi Paradox is, "tripped on shoe laces while scoring +10 on the sosh meeds".

You don’t know that so why bother speculating like you do ?

Are you really saying that people shouldn’t use their imagination and not talk or think about things they are not 100% sure of?

Because if that’s what we did then no need to worry about getting back to the Stone Age, we would have never left it anyway... Man’s ability to dream, imagine and explore the unknown both physically and intellectually is what got us where we are.

"Human civilization will die an embarrassing death."

The language used here isn't speculation, it's absolute. If the author was to prefix the statement using speculative terms then I wouldn't have said a thing.

The remark came across as highly negative, unsubstantiated and not useful so I called it out.

Edit: Downvotes? You're going to have to do better than that before I change my mind.

The remark came across as highly negative

Probably because it was.

It was also a reply to a comment about a hypothetical future, which sort of implies speculation for all child comments.

And how can something be absolutely embarrassing? What is and is not embarrassing is an opinion. How would someone word that in a way that fits your requirements?

The comment doesn’t need to fit my requirements, it surprises me the Internet now feels like it needs to convince people to think a certain way.

Dr Strangelove, Skynet in Terminator, all of this has been talked about many times before. The truth is that already most of nukes are probably automated to ensure the concept of mutually assured destruction works.

But really, as long as everyone believes in MAD, there is really no reason to carry it out. Engineers working on these systems could quite reasonably, and in good faith, install break points and avoid meaningless loss of life. Then the only MAD system which actually needs implementation is if one of those engineers talks..

But for everyone to believe it, it has to be an at least reasonable belief. Luckily, that's a low-ish bar with fairies being a somewhat common belief too (I have no numbers but want them...). So those engineers for those same good faith reasons install redundancies. Heh, it's a balance that may or may not tip positive (everything working) or neutral (nothing happens) or negative (everything sabotaged by an enemy actor) and we'll only know after the trigger has been pulled.

You don't even have to automate all that and still achieve the goal of destroying both the human civilization as we know it: it was already existing in practice, simply by delegating (human) responsibilities and effective since the thermonuclear bombs were stockpiled. It's documented in the book: https://www.bloomsbury.com/us/the-doomsday-machine-978160819...

> The truth is that already most of nukes are probably automated to ensure the concept of mutually assured destruction works.

Why do people say things like this? They aren't. There is still a human-in-the-loop for all systems I'm aware of.

The problems with nukes are that (a) the command and control systems can't reliably prevent one or a small number of people from going from going rogue or (b) the 'human' in the loop has to make a quick decision whether to launch based on potentially crappy inputs.

The latter case has happened enough in the past that it should be pretty clear it's a serious problem. See https://en.wikipedia.org/wiki/List_of_nuclear_close_calls for a list.

Pretty sure they took the "men out of the loop" back in 1983 ... https://www.youtube.com/watch?v=5bF1_PGMAj0

"Or launching nuclear weapons based on the advice of AI"

That's the crazy thing about hypersonic nuclear weapons... A bomb going from Russia to America in under 5 minutes, who else is going to decide than an algorithm... Knowing world peace is partly ensured by mutually assured destruction :s

(To be clear, I'm for a moratorium on hypersonic nuclear warfare, for sure.)

If we play fast and loose with the definition of AI we've come close already with operation RYAN (https://en.wikipedia.org/wiki/RYAN):

> By May of 1986, these binders had evolved into a catalogue of 292 indicators of “signs of tension.” - https://www.wilsoncenter.org/publication/forecasting-nuclear...

I'm not sure how much was computerized but it basically took a bunch of signals and fed them into an algorithm to tell if the US was about to nuke them.

Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion of Safety is a great book about how close you can get to launching a war unintentionally.

It is also a chilling documentary on Prime and probably YouTube. Just to think how close we came to losing part of Arkansas and Tennessee...and then to find out how common this was.

https://www.google.ca/amp/s/www.globalresearch.ca/strangelov...

Just wait until somebody thinks it's a good idea to declare war, based on the advice of someone in their cabinet

Maybe I'm a naive optimist but I doubt that will ever happen (or until we have AGI). We as a species dont really like to give up control. See for example the "Auto Pilot": we have the tech to fly planes safer than human pilots for decades now, yet we don't because we don't like the feeling of not being in control. I can't imagine that a group of people decide to hand control of a WMD to a computer.

It may also be the need to blame someone, the pilot in this case, in case of a disaster.

We've almost had several nuclear weapons launched due to flaws in human decision making, two that were launched with the inputs of humans, and a ton of wars launched with the input of humans. What about an AI being involved automatically makes the situation so much worse?

We understand how people work and think pretty well, their edgecases, and countermeasures to said edgecases.

We don't understand edge cases of AI much at all, and the terrain-space of "possible minds" they might be is vastly broader than that of a human mind, and utterly outside our experience.

An AI might work with absolute 100% flawless precision for 100 years, then decided one day to kill everyone. A lot harder to see that coming from a totally alien intelligence than a human one. AI does not think like humans, have human values, or care about what humans care about, that might make them extremely unpredictable and dangerous.

There will be no time for iterative development, or learning from our mistakes, a sufficiently advanced AI knows we would try to stop it, and make sure that we never can.

https://www.youtube.com/watch?v=tcdVC4e6EV4

Look at any of the "flash crashes" of recent years were out of control AI escalated the problem much faster than humans could react to it. Just take that and imagine that there are nuclear weapons involved.

Ok. None of those have had any long term negative outcomes. What was I supposed to imagine?

A stock dipping isn’t so bad because it can go back up. A nuclear missiles cannot unexplode.

And professionals price the downside into those trades. You don’t engineer trades the same way you do nukes cause it literally doesn’t matter if you get them wrong. As the history of “flash crashes” has shown.

A Bit easier to recover from a market dip than a nuclear strike don't you think?

It wasn't even that you recovered from the dip, the trades were actually busted (invalidated.)

Can't unexplode a bomb, for sure.

Trades are not routinely busted during flash crashes.

They were in "the" flash crash.

It had some pretty long term negative effects for Knight Capital.

Knight didn’t suffer from a flash crash. They suffered from a operational risk incident. This is literally something that professionals price into their trades. The long term negative ramifications were nothing. The next day a new firm was running the same trades under the same name.

I thought Knight got bought out a year later?

And for those few failures think of the trillions and more of the decisions that machines made that did not harm, but made lives better than before machines were doing work for us.

Reply on news.ycombinator.com