Hacker News Reader

It's also contingent upon giving the AI complete control over an army of robots and telling it to do whatever it feels like while we completely ignore what it's doing (I think Bostrom's argument is that it also has the magical ability to exponentially increase it's intelligence in a very short time period). Like most of the AI-apocalypse, it assumes that a string of highly improbable things would all occur at the same time, and then says, "wouldn't that be terrible?"

Well, yeah. But CERN creating a killer black hole would also be terrible. But we should think about what's probable, not just what's scary.

That doesn't follow at all. A sufficiently smart AI could do all sorts of obtuse things to achieve its aims. It could speculate on the stock market or go on an identity theft spree to pay for more servers. It could start a business and hire employees to run a robot factory. It could start a propaganda campaign to persuade humans to do its bidding. It could lobby politicians to remove legal impediments.

The fundamental principle of the paperclip argument is that the motivations of an AI will not necessarily align with our interests. An AI may do all sorts of things that seem nonsensical or morally repugnant to human beings if it does not share our moral intuitions.

If the intelligence of that AI significantly exceeds the range of human intelligence, we may be powerless to stop it or even to comprehend what it is doing. A rogue AI could become a catastrophically nasty Stuxnet, distributing itself across the billions of Turing machines we have networked together. Our only effective response may amount to "erase every data storage device in the world and start from scratch".

Yes, and the apocalypse resulting from that mismatch of goals requires the things the parent post talks about (humans ignoring this AI and blithely giving it free reign without testing it)

> "don't turn humans into paperclips" is part of the context of "make more paperclips"

The idea in this thought experiment is not that the AI parses this sentence in the context of human culture (then it would likely comprehend that the actual intention is to maximize the economic success and eventually the human preferences of its creator). What is meant is that the objective is crudely implanted into the AI as an ultimate goal, in a similar way to how sustenance, pain avoidance and affiliation are very basic goals in our cognitive system. It is not entirely obvious that this is a stupid thing to do; hence the thought experiment. Will the AI suppress its urge once it comprehends human culture enough to understand the intentions of its creator? Will it rather successfully learn all the tricks to convert matter into paperclips before it considers studying human values? If the AI does not have a curiosity objective, it will likely not care about us very much, apart from the information that helps it optimizing its objective function.

I think it is way more likely we have AI that understands human context and values before we have AI that is super intelligent.

What makes you so certain?

Also, https://en.wikipedia.org/wiki/Intermediate_value_theorem

I never said am certain, I'd assign both cases about the same probability. Since I've already stated why I think the malevolent simple AI scenario such as a paperclip maximizer is plausible, you haven't stated why you find it implausible. That is why I asked.

Also, what does the Intermediate value theorem have to do with it?

You asked why I was certain. So I asked why you were certain. The onus is on the people putting out apocalyptic theories to justify the apocalyptic scenario (extraordinary claims require extraordinary evidence - a central tenet of the scientific method).

We are going to have a normal-intelligent AI before a superintelligent AI.

> If it isn't smart enough to see that "don't turn humans into paperclips" is part of the context of "make more paperclips"

It'll see, but it won't care. See http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/

Why not? A major problem with AI apocalypse arguments is that they are quite vague and terms are poorly defined. One thing that AI apocalypse believers often talk about is the danger of self improving AI. But I think AI won't be able to self improve until it is good enough such that it knows and cares about context. Self-improvement will require such s comprehensive understanding of the goal that AI will need to understand these concepts.

You might note that my argument is pure speculation without much basis in evidence. This is intentional, because it's all the arguments I've seen expounding on the AI apocalypse are equally speculative.

Intelligence doesn't work like that. Take AIXI, define a reward in terms of paperclips and (if it were actually computable) you will kill everything.

Intelligence and any consideration of its goals at all can be completely unrelated.

you are making the assumption that a "smart enough" AI will intrinsicly value human life.

There is no reason for this to be the case. What something values is almost entirely independant of its intelligence.

At best, a human would be worthless to an AI who 100% only values paperclips (better as paperclips than humans), at worst seen as a threat to its ability to continue. (better to kill them off ASAP before try to stop me)

Thats the entire point of the paperclip maximizer thought experiment, to demonstrate that values, especially values humans care about, are not the same thing as inteligence.

Your entire ability to not "take things literally" is based off of a large amount of built in assumptions about whats important, your human nature and cultural experience. An AI would lack any of that, and if it was programmed only to value paperclips, even if was the most inteligent being in existence, thats all it would value, to the exclusion of everything else.

Consider autistic people with very high IQs, and due to their condition, very little understanding/concern about the emotions of others.

But autistic people are not dominating other humans with their "high IQs." In fact most people with autism struggle to lead a productive life and require quite a bit of social assistance to do so.

No one is doubting that machines fail to understand human values. The doubt is that such a machine could dominate its environment, or even work independently without careful management by humans.

The papercliper would know you didn't program it the way you intended to, but it wouldn't care. That wouldn't bring about more paperclips.

A human might know the ultimate goal of sex is to reproduce, but still choose to use contraceptives. It's defeating the purpose of the original goal and just pursuing what feels good.

The same thing could possibly (likely?) happen with an AI system. While it might be able to reason about the intent of it's design (what the code's supposed to do), it'll still pursue what "feels good", which is an artifact of it's actual design (what the code actually does, bugs and all...).

I'm imagining a self-aware AI doing good all day then retiring to it's private realm to watch videos of paper-clip manufacturing.

This is one of those jokes too funny to keep to yourself and too contextual to tell other people.

Sex is how humans reproduce, but it's a pretty big (and highly metaphysical) stretch to say that that is the "ultimate goal" of sex. As far as we can tell, sex and reproduction have no ultimate goals at all--they just are. We have some idea of how they evolved over billions of year, but we have no idea how non-life became life, or why.

We're into some pretty deep stuff here. But the point is that no master creator has commanded us to reproduce infinitely and spare no expense. In fact our increasingly conscious control of our reproduction is reducing, not increasing, our risk of a catastrophic population explosion.