Hacker News Reader

Comment by jononor | Hacker News Reader

Check for instance the 'Paperclip maximizer' thought experiment, https://wiki.lesswrong.com/wiki/Paperclip_maximizer

Yeah the problem is where would you put in the command to maximize paperclips? Even though deep learning is very powerful, it doesn't contain a utility function. The utility function to maximize paperclips would be some kind of module separate from a set of deep learning nets that would coordinate and read inputs from all the nets and then somehow give commands to some actuators somewhere (which it would also need intelligence about). All those deep learning networks would have to already be tuned to be very accurate in their own right before they are hooked up to any kind of controller.

That was my point that, AI has been confused with the deep learning gains in the past years, even though, the hard part about how to coordinate all those inputs and also give the right commands to actuators is still not near to solved, much less a completely virtual superintelligence that has its own "virtual" goals and virtual simulations of both its inputs and potential actuators in the real world. This would be akin to a kind of intelligence algorithm, and not deep learning vector algorithms that reveal structures in data.

The Paperclip maximizer assumes that you could have an AI smart enough to turn the world into paperclips, but not smart enough to understand what the human meant by "make me some paperclips".

My guess is that paperclipping the world takes magnitudes more intelligence than understanding what a human means when saying something ambiguous.

Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you told it to care about something else.

For example, maybe you inadvertently programmed it to "be totally literally 100% certain that you've completed the task I told you to do". Then from its perspective, there's always a tiny, tiny chance that its sensors are being fooled or malfunctioning, so it can't be literally 100% certain that it's ever made a paperclip successfully, so by your programming, it should keep making more paperclips. This is independent of whether or not you wanted it to do that: it's a result of what you programmed it to do and what you told it to do.

If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished.

We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.

I did indeed say that a superintelligent AI could realise that we don't want to paperclip the world. But it need not care what we want, unless we're really, really careful about how we program it.

What is "ridiculous" about continuing a task because we're not certain that it's done yet? It's only your human moral system saying that. Just because humans usually don't value things enough to pursue them to the exclusion of all else…

I tell my super intelligent assistant, who's much better at understanding human language than Siri, to make me some paperclips.

It understands some to mean more than one and less than infinity. In fact, some means less than "a lot". The meaning of a lot depends on the context, which happens to be paperclips for me.

What is "some" paperclips for me? It depends on how many papers I might need to clip (or whatever use I might have for paperclips). My super intelligent assistant would be able to work out a good estimate.

After having an estimate, it can go make me "some" paperclips, and then stop somewhere short of paper clipping the entire world.

Alternatively, it could just ask me how many "some" means.

- Hill Climbing - Local Maximum - Gradient Descent

I don't pretend to have any sort of expertise in these sorts of discussions, so I thought I would throw out some easily wikipedia'd terms that seem to back your thoughts regarding convergence vs. exponentiation.

You're still assuming that the assistant cares about doing what I mean, rather than doing precisely what I say or doing what I programmed it to do. That only happens if we're really, really careful about how we programmed the AI in the first place. I grant you 100% that the AI probably knows precisely how many paperclips I want, but you're assuming that it wants the same number of paperclips that I want.

What an agent considers to be "good" is orthogonal to how intelligent that agent is. An agent of arbitrary intelligence can have arbitrary goals; the goals of an intelligent agent need not in principle look anything like those of a human. The only reason a superintelligent AI's goals would look like those of a human is because the humans very, very carefully programmed them into the AI somehow. Very, very careful programming is not a feature of how humans currently approach AI research.

No. It means you could program an AI to a) make paperclips and b) increase its own intelligence. Assume you program (a) incorrectly (in the sense of lacking the appropriate safeguards about when not to make more paperclips) but (b) correctly. Then, it doesn't matter that the AI will eventually understand what you want perfectly, because what it will want to do is to keep making paperclips. The whole argument is that if you have a self-improving agent with a goal and the ability to improve in ways you can't predict at all, then you better make sure its goal matches yours.

Of course, the obvious solution is: "make the objective function equivalent to 'do what I want you to do'", but the problem is we might not know how to encode that without help from a super-AI.

Even "do what I mean" is not obviously correct. Yudkowsky used to advocate the idea of "coherent extrapolated volition": the idea of what I would want the AI to do, were I much more intelligent, knew more, could think faster, were more how I wished I were, and so on.

"Do what I mean" is correct in that it reduces super-human AI to the same category of other human technologies: as good or as bad as whatever we choose to do with them. "Coherent extrapolated volition" might be preferable, but if we could get to the first, we could then ask for the second.

I don't think the intelligence aspect is the biggest problem with that paperclip maximiser concept. It's the physical ability to actually turn stuff into paperclips. Or whatever else the nearest analogy would be.

Doesn't matter how determined the machine is to make them, the rest of the world won't exactly lay down and let it. Nor would its goals often be possible with the amount of resources available in its surroundings.

It's basically the cat problems from this tongue in cheek article:

http://idlewords.com/talks/superintelligence.htm

Reply on news.ycombinator.com