Comment by lowbloodsugar | Hacker News Reader

Comment by lowbloodsugar | Hacker News Reader

Parent Full thread

>It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking.

"Guns don't kill people, AIs kill people" is where we are going, I think. This is the discussion: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

The discussion is not about a mathematical representation of AI. The discussion is about the actual implementation of AI on physical computing infrastructure which is accessible by at least one human on planet earth.

The credible danger, argued in various places, including superintelligence by Nick Bostrom, is that the "system under review" here is "every physical system on planet earth" because an AI could gain access to whatever systems exist on said planet, including human minds (see "Nazis").

So much as we might discuss the problems of letting a madman get control of the US, Russian, UK, French or Chinese nuclear arsenals, we might discuss the problem of building an AI if the act of building the AI could result in it taking over the nuclear arsenals of those countries and using it against humans. That takeover might involve convincing a human it should do it.

I don't understand this argument (the "Terminator" scenario). AI could conceivably self replicate and evolve as software but it still needs hardware to run on, power, networking etc etc. There's no better way to kill that than to start a nuclear war or engineer some super virus that kills everyone.

It's hard to see any scenario where AI could become a dominant force without significant human collaboration. Perhaps somewhere like North Korea where a very small elite has complete control over the the population it could happen but it sounds a lot like sci-fi to me. I'd love to hear some plausible scenarios for the counter-argument. I've seen a lot of "I think there's an x% chance we're in trouble" arguments which might be convincing for job losses, but I don't find at all plausible as a case for human extinction or indentured servitude (the "Matrix" scenario).

Sure. I recommend reading superintelligence by Nick Bostrom.

But I think the key failure here, in your thinking, is that you can't conceive of something, and that therefore something can't happen, and you're doing that in the context of things that are smarter than you. You being able to conceive of it is simply not required for it to happen.

Also, Colossus: The Forbin Project was a great movie from 1970 on this subject. Spoilers: an AI takes control of the nuclear arsenal and then threatens humans with extinction if they do not serve it. The humans do serve it, of course, because the humans in charge don't want to die, and are entirely fine with enslaving the rest of us.

The book superintelligence by Nick Bostrom gets into the fine details of all the different ways an AI would escape, why it would escape, and why it wouldn't take a chance with a species that murders its own kind for fun and profit.

Superintelligence and Life 3.0 seem to come up as recurring references in the discussion. I've only read synopses of both but frankly I find the argument that melevolent AI "escape" could occur without being noticed and thwarted a bit far fetched.

There's a good counterargument here[1] that seems reasonable: "One of the features of intelligence explosion that most preoccupies Bostrom and Yudkowsky is that it’s not a problem that we get to have many attempts at. In the Terminator movies, humans don’t get to approach a newly self-aware Skynet and request a do over. One minute Skynet is uncomplainingly complying with all human directives. The next, it’s nuking us. I suspect that we are likely to have plenty of opportunities for do overs in our attempts to make autonomous AIs. Autonomy is not an all-or-nothing proposition. The first machine agents are likely to be quite clumsy. They may be capable of forming goals in respect of their world but they won’t be particularly effective at implementing them. This gives us plenty of opportunity to tweak their programming as they travel the path from clumsy to sophisticated agency"

[1] https://jetpress.org/v26.1/agar.htm

How would you know if it is a good counter argument if you haven’t bothered to read the argument?

Ok - fair comment. So I found a copy of Superintelligence. The argument is pulp sci-fi at best: "The final phase begins when the AI has gained sufficient strength to obviate the need for secrecy. The AI can now directly implement its objectives on a full scale. The overt implementation phase might start with a “strike” in which the AI eliminates the human species and any automatic systems humans have created that could offer intelligent opposition to the execution of the AI’s plans. This could be achieved through the activation of some advanced weapons system that the AI has perfected using its technology research superpower and covertly deployed in the covert preparation phase. If the weapon uses self-replicating biotechnology or nano-technology, the initial stockpile needed for global coverage could be microscopic: a single replicating entity would be enough to start the process. In order to ensure a sudden and uniform effect, the initial stock of the replicator might have been deployed or allowed to diffuse worldwide at an extremely low, undetectable con-centration. At a pre-set time, nanofactories producing nerve gas or target- seeking mosquito-like robots might then burgeon forth simultaneously from every square meter of the globe (although more effective ways of killing could probably be devised by a machine with the technology research superpower)."

Look - nothing's impossible, but I agree with the counter argument that an advanced AI still starts as a "brain in a vat", with no experience of agency in the physical world. In order to successfully take over you have to assume it can develop all the physical world capability it needs, in secret and get it right first time. That seems implausible.

Reply on news.ycombinator.com