Comment by joe_the_user | Hacker News Reader

Comment by joe_the_user | Hacker News Reader

Parent Full thread

What is meant is that the objective [paper-clip-building] is crudely implanted into the AI as an ultimate goal, in a similar way to how sustenance, pain avoidance and affiliation are very basic goals in our cognitive system.

Sure but it is a continuation of the argument in different forms.

If one is saying the AI is just a combination of crude imperatives surrounded by "intelligence" then consider, could you build a human-level AI without that AI speaking as a person and without that AI having digested culture?

Even the neural networks that exist today are rather dependent on their "training sets". Watson is a glorified natural-language interface to wikipedia and related sources.

Many hypothetical-AI arguments that appear, oppositely, imagine that some omnipotent thing will be created without that creation following the obvious path of digesting that vast store of information and communication that is human knowledge.

However, you might be saying that a creator first gives the AI all this nuanced understanding of human knowledge and communication but then doesn't do the obvious thing, give it the imperative, "follow my directives as a loyal but intelligent servant would" and says incidently, "one directive is get me some paper clips". Rather, the creator say "now that you understand everything, make paper clips ruthless, beyond all else, all other actions build to up to this paper clip building thing". Sure, GAIs could do insane damage but it I think one if one considers that GAIs would not be produced by accident, such damage almost certainly be a product of human intentions.

What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives. So far, it appears that a set of objectives is in fact necessary for intelligence.

The question is whether the AI will go crazy like a mentally ill person, if it lacks empathy and curiosity. It may seem intuitive that a superintelligent AI will understand our values (since it is superintelligent), but, assuming intelligence is necessarily an optimization process of predefined goals, why would it be interested in us in the slightest, if we don't pose an advantage for it optimizing its objectives (e.g. sustenance)? Worse, we might be in its way because we could end up competing with it for resources such as sunlight, carbon compounds and oxygen.

What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives.

No, it's pretty easy. I predict a super intelligence wouldn't evolve at all from a set of simple objectives.

(that may seem a little snarky but the original question has the implied assumption that super inteilligence could evolve from only a set of simple objectives and I think that assumption is implausible, is only accepted because it's made implicitly, etc)

But I guess that's a fundamental disagreement. I think it's relatively "obvious" that creating an intelligent system would involve the intentional crafting as well as inputs of particular immediate goals. If intelligence could from just whatever system gets complex and has feedback loops, then I'd agree we're in trouble and need to go around smash all the high-end thermostats in a fashion akin to a b-grade horror movie. But clearly I think that's a dubious "if".

I made that assumption explicitly, actually. To make it worse, it seems plausible that these kinds of dangerous AIs will be the most simple ones to build, since they won't require a lot of goal engineering.

Intelligence is a superset of feedback loops. It is concerned with cases in which feedback loops are not sufficient to optimize the agents objectives, and attempts to reach them using learning and prediction. The complexity in behavior comes from interacting with a complex environment and having many model parameters; not (necessarily) from the initial goals. (As an intuition pump, have a look at GoogleMind's atari reinforcement learner. The essential parts of the code fit onto a single page [1].)

[1] https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner

Reply on news.ycombinator.com