Hacker News Reader

Comment by Analemma_ | Hacker News Reader

When you want to discuss "the myth of a superhuman AI", it's important to carefully separate the two categories of claims:

1. The claims by economists that AI-- even if it's not "strong AI"-- will put lots of people out of a job with potentially severe societal/economic repercussions

2. The claims by the Bostrom/Yudkowsky/etc. crowd that an AI intelligence explosion will cause the extinction of humanity

Without saying anything about the plausibility or lack thereof of either 1 or 2, I think we can all agree that they are very different claims and need to be analyzed separately. Right from the very first sentence the author seems to muddle the two, so I don't think there's much of cogent analysis in here.

View on HN

> Right from the very first sentence the author seems to muddle the two, so I don't think there's much of cogent analysis in here.

If you read past the first sentence, it's quite obvious that the article is mostly concerned with claims of your type 2). Also I'm not sure why it would be so important to carefully separate the two categories of claims - they both rely on the assumption that AI can perform a wide variety of cognitive tasks better and cheaper than humans, which is addressed at length and cogently in the article. Also I believe that the economists you mention are concerned more about robotics than about AI per se, so I'm not even sure why you've brought in that point.

> 2. The claims by the Bostrom/Yudkowsky/etc. crowd that an AI intelligence explosion will cause the extinction of humanity

Maybe someone can correct me if I'm wrong here, but I have a hard time understanding what /any/ "utility function" would be, that the super ai people talk about. It can't be a passive deep learning network that parses information and gives an output, it has to be some kind of complex perception / action loop of many neural nets and actuators in the real world that somehow leads to an intelligent self improving behavior? I guess you could make a deep learning controller for self driving cars say, and if an input to many cars is wrong, all the cars crash and create a big cascading mess of wrong input values, but that kind of accident is a far cry from an intelligent chain of events where every link in the chain is an intelligent decision but the ultimate goal is bad.

And, do we even know any way to chain many deep learning networks together that accurately give correct output values, that we then can hook up to a controller to give a utility function, which can then lead to a cascade of intelligent decisions across domains?

Check for instance the 'Paperclip maximizer' thought experiment, https://wiki.lesswrong.com/wiki/Paperclip_maximizer

The Paperclip maximizer assumes that you could have an AI smart enough to turn the world into paperclips, but not smart enough to understand what the human meant by "make me some paperclips".

My guess is that paperclipping the world takes magnitudes more intelligence than understanding what a human means when saying something ambiguous.

Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you told it to care about something else.

For example, maybe you inadvertently programmed it to "be totally literally 100% certain that you've completed the task I told you to do". Then from its perspective, there's always a tiny, tiny chance that its sensors are being fooled or malfunctioning, so it can't be literally 100% certain that it's ever made a paperclip successfully, so by your programming, it should keep making more paperclips. This is independent of whether or not you wanted it to do that: it's a result of what you programmed it to do and what you told it to do.

If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished.

We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.

I did indeed say that a superintelligent AI could realise that we don't want to paperclip the world. But it need not care what we want, unless we're really, really careful about how we program it.

What is "ridiculous" about continuing a task because we're not certain that it's done yet? It's only your human moral system saying that. Just because humans usually don't value things enough to pursue them to the exclusion of all else…

I tell my super intelligent assistant, who's much better at understanding human language than Siri, to make me some paperclips.

It understands some to mean more than one and less than infinity. In fact, some means less than "a lot". The meaning of a lot depends on the context, which happens to be paperclips for me.

What is "some" paperclips for me? It depends on how many papers I might need to clip (or whatever use I might have for paperclips). My super intelligent assistant would be able to work out a good estimate.

After having an estimate, it can go make me "some" paperclips, and then stop somewhere short of paper clipping the entire world.

Alternatively, it could just ask me how many "some" means.

- Hill Climbing - Local Maximum - Gradient Descent

I don't pretend to have any sort of expertise in these sorts of discussions, so I thought I would throw out some easily wikipedia'd terms that seem to back your thoughts regarding convergence vs. exponentiation.

You're still assuming that the assistant cares about doing what I mean, rather than doing precisely what I say or doing what I programmed it to do. That only happens if we're really, really careful about how we programmed the AI in the first place. I grant you 100% that the AI probably knows precisely how many paperclips I want, but you're assuming that it wants the same number of paperclips that I want.

What an agent considers to be "good" is orthogonal to how intelligent that agent is. An agent of arbitrary intelligence can have arbitrary goals; the goals of an intelligent agent need not in principle look anything like those of a human. The only reason a superintelligent AI's goals would look like those of a human is because the humans very, very carefully programmed them into the AI somehow. Very, very careful programming is not a feature of how humans currently approach AI research.

No. It means you could program an AI to a) make paperclips and b) increase its own intelligence. Assume you program (a) incorrectly (in the sense of lacking the appropriate safeguards about when not to make more paperclips) but (b) correctly. Then, it doesn't matter that the AI will eventually understand what you want perfectly, because what it will want to do is to keep making paperclips. The whole argument is that if you have a self-improving agent with a goal and the ability to improve in ways you can't predict at all, then you better make sure its goal matches yours.

Of course, the obvious solution is: "make the objective function equivalent to 'do what I want you to do'", but the problem is we might not know how to encode that without help from a super-AI.

Even "do what I mean" is not obviously correct. Yudkowsky used to advocate the idea of "coherent extrapolated volition": the idea of what I would want the AI to do, were I much more intelligent, knew more, could think faster, were more how I wished I were, and so on.

"Do what I mean" is correct in that it reduces super-human AI to the same category of other human technologies: as good or as bad as whatever we choose to do with them. "Coherent extrapolated volition" might be preferable, but if we could get to the first, we could then ask for the second.

I don't think the intelligence aspect is the biggest problem with that paperclip maximiser concept. It's the physical ability to actually turn stuff into paperclips. Or whatever else the nearest analogy would be.

Doesn't matter how determined the machine is to make them, the rest of the world won't exactly lay down and let it. Nor would its goals often be possible with the amount of resources available in its surroundings.

It's basically the cat problems from this tongue in cheek article:

http://idlewords.com/talks/superintelligence.htm

Yeah the problem is where would you put in the command to maximize paperclips? Even though deep learning is very powerful, it doesn't contain a utility function. The utility function to maximize paperclips would be some kind of module separate from a set of deep learning nets that would coordinate and read inputs from all the nets and then somehow give commands to some actuators somewhere (which it would also need intelligence about). All those deep learning networks would have to already be tuned to be very accurate in their own right before they are hooked up to any kind of controller.

That was my point that, AI has been confused with the deep learning gains in the past years, even though, the hard part about how to coordinate all those inputs and also give the right commands to actuators is still not near to solved, much less a completely virtual superintelligence that has its own "virtual" goals and virtual simulations of both its inputs and potential actuators in the real world. This would be akin to a kind of intelligence algorithm, and not deep learning vector algorithms that reveal structures in data.

We don't know how to do this yet, but humans have things which are a bit like utility functions, "built in" in a way we don't understand. However it's done, it leads to self-improving behaviour in humans (many humans value learning and becoming better at things). Just because we don't know how to do it with deep learning and/or neural networks doesn't mean that it's a) not possible with those techniques, and b) that it's not possible with some other technique. (When a neural network becomes sophisticated enough to model its own workings and its relation to the environment, I don't see a knockdown argument for why it couldn't in principle optimise itself, given help from the outside.)

You make a good point, but it seemed pretty clear to me that the author was mostly discussing the (2) scenario as you labeled them. To me, his analysis was very cogent, considered within the broader framework of how we ultimately "just don't know" about some of this stuff.

Reply on news.ycombinator.com