Hacker News Reader

Comment by sawwit | Hacker News Reader

What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives. So far, it appears that a set of objectives is in fact necessary for intelligence.

The question is whether the AI will go crazy like a mentally ill person, if it lacks empathy and curiosity. It may seem intuitive that a superintelligent AI will understand our values (since it is superintelligent), but, assuming intelligence is necessarily an optimization process of predefined goals, why would it be interested in us in the slightest, if we don't pose an advantage for it optimizing its objectives (e.g. sustenance)? Worse, we might be in its way because we could end up competing with it for resources such as sunlight, carbon compounds and oxygen.

View on HN

What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives.

No, it's pretty easy. I predict a super intelligence wouldn't evolve at all from a set of simple objectives.

(that may seem a little snarky but the original question has the implied assumption that super inteilligence could evolve from only a set of simple objectives and I think that assumption is implausible, is only accepted because it's made implicitly, etc)

But I guess that's a fundamental disagreement. I think it's relatively "obvious" that creating an intelligent system would involve the intentional crafting as well as inputs of particular immediate goals. If intelligence could from just whatever system gets complex and has feedback loops, then I'd agree we're in trouble and need to go around smash all the high-end thermostats in a fashion akin to a b-grade horror movie. But clearly I think that's a dubious "if".

I made that assumption explicitly, actually. To make it worse, it seems plausible that these kinds of dangerous AIs will be the most simple ones to build, since they won't require a lot of goal engineering.

Intelligence is a superset of feedback loops. It is concerned with cases in which feedback loops are not sufficient to optimize the agents objectives, and attempts to reach them using learning and prediction. The complexity in behavior comes from interacting with a complex environment and having many model parameters; not (necessarily) from the initial goals. (As an intuition pump, have a look at GoogleMind's atari reinforcement learner. The essential parts of the code fit onto a single page [1].)

[1] https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner

Reply on news.ycombinator.com