Hacker News Reader

Comment by rnk | Hacker News Reader

What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist? Obviously they are dangerous for the world because we might kill everyone if someone makes a mistake or gets mad and shoots one first. The danger of nukes is more obvious than why AI is dangerous.

View on HN

> "What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist?"

The Wikipedia article on mutual assured destruction (https://en.wikipedia.org/wiki/Mutual_assured_destruction) traces the origin and evolution of exactly that analysis, including even pre-atom bomb precursors based on earlier weapons that would hypothetically make war "too terrible to ever happen again".

What Chalmers is quoting is a tweet from Yudkowsky claiming that “ruin” is the guaranteed result of inventing AGI.

If someone said “the invention of nuclear weapons is guaranteed to lead to global destruction,” you might ask to see their reasoning for such a strong assertion. (As opposed to your assessment of nuclear weapons as a risk to be managed.)

Well THAT support starts with the experimental records of Manhattan Project and fills a whole library. Support for current ML techniques being dangerous in that sense are handwavy descriptions of what might well be science fiction for another 500 years.

Well, support for ML techniques being stupidly wrong is much bigger. It's the default state.

Then there's some less support for them being misleadingly wrong, much more rarely.

Science fiction Terminators it is not. Paperclip maximizers, however, yes. In fact it can be argued that a typical capitalist company is a variant of a paperclip maximizer and is a simple form of superhuman intelligence.

Nick Bostrom invented that thought experiment. Start from there and go digging through AI safety research of which there are a few journals.

Real life samples include 2010 flash crash, social media automatically optimizing for engagement at any cost, a few computer vision issues causing medical misdiagnosis, Tesla's "self-driving", video game AI exploiting engine bugs rather than playing, suddenly racist face recognition...

And those are just mistakes, a malicious actor can do more with these. DARPA and IARPA have a whole damn wing on handling trojan attacks against ML systems.

A superhuman, even weakly superhuman paperclip maximizer is an extremely dangerous thing. (Analyze climate change in this light...)

Sorry, I don't understand. Are you saying that AI is of the same sort of danger as nukes?

"The danger of nukes is more obvious than why AI is dangerous."

Succinctly stated. This is why people are asking for canonical arguments for AI Ruin.

First, I was surprised about the idea that there must be an exact description of dangers before someone can investigate the dangers and consider them. So I was raising the issue of nukes as obviously dangerous, ai less so, we can consider the dangers of AI or anything in the abstract.

I do see AI as very dangerous though. My personal belief is we'll build AGI, every country will have some of their own. Now consider the next war US versus China say or Russia, people will be doing everything possible to win. Eventually we'll make AI controlled robots or weapons that don't have human controls.

There's a less dangerous, but still incredibly destructive problem of AI, propaganda, fake information, photos, etc, becoming ever more capable of making influential damaging fake news. The US is already suffering under this without AI, 1/3 of the people believe things that aren't true about the election, science, etc. Multiply that by 100 and a worse problem occurs.

> First, I was surprised about the idea that there must be an exact description of dangers before someone can investigate the dangers and consider them.

Of course there has to be a description of dangers before someone can consider them, otherwise how do we evaluate risk? How else do we know whether we are overreacting? It's not like the "ruin" people suddenly woke up yesterday morning in a panic. They have been talking about this for years. In all that time, there has been no dispassionate but thorough analysis of the dangers?

What about the Non Proliferation Treaty?

The nuclear non-proliferation treaty has somewhat limited nukes. But now Iran is on the verge, Israel, India, Pakistan , North Korea and other countries must be pursuing them. North Korea has shown the value of them because nobody can attack them or stop them because they do have some nukes and missiles. Israel and Iran are always attacking each other and regardless of the reason or goodness of either side, I'd pick them as the two most likely countries to start a nuclear war. Maybe even worse than Pakistan versus India.

I'd say we need a new strategy on top of the nuclear non-proliferation treaty.

Reply on news.ycombinator.com