Comment by fchollet | Hacker News Reader

Comment by fchollet | Hacker News Reader

Here is a short recap of what the people who understand where machine intelligence is now, and where it is going, think of the whole "evil superintelligent AI" blogging trend we're seeing these days. These are some of the people who created the field. In particular LeCun and Bengio are in part responsible for the recent renewed interested in it ("Deep Learning").

Yann LeCun: "Some people have asked what would prevent a hypothetical super-intelligent autonomous benevolent A.I. to “reprogram” itself and remove its built-in safeguards against getting rid of humans. Most of these people are not themselves A.I. researchers, or even computer scientists.[...] There is no truth to that perspective if we consider the current A.I. research. Most people do not realize how primitive the systems we build are, and unfortunately, many journalists (and some scientists) propagate a fear of A.I. which is completely out of proportion with reality. We would be baffled if we could build machines that would have the intelligence of a mouse in the near future, but we are far even from that." http://www.popsci.com/bill-gates-fears-ai-ai-researchers-kno...

Yoshua Bengio: "What people in my field do worry about is the fear-mongering that is happening [...] As researchers, we have an obligation to educate the public about the difference between Hollywood and reality."http://www.popsci.com/why-artificial-intelligence-will-not-o...

Rodney Brooks: "I say relax everybody. If we are spectacularly lucky we’ll have AI over the next thirty years with the intentionality of a lizard, and robots using that AI will be useful tools. And they probably won’t really be aware of us in any serious way. Worrying about AI that will be intentionally evil to us is pure fear mongering. And an immense waste of time."http://www.rethinkrobotics.com/artificial-intelligence-tool-...

Michael Littman: "Let's get one thing straight: A world in which humans are enslaved or destroyed by superintelligent machines of our own creation is purely science fiction. Like every other technology, AI has risks and benefits, but we cannot let fear dominate the conversation or guide AI research." http://www.livescience.com/49625-robots-will-not-conquer-hum...

Expert opinion is not unified on this. E.g.,

Stuart Russell: "A highly capable decision maker – especially one connected through the Internet to all the world's information and billions of screens and most of our infrastructure – can have an irreversible impact on humanity ... We need to build intelligence that is provably aligned with human values ... This issue is an intrinsic part of AI, much as containment is an intrinsic part of modern nuclear fusion research." http://edge.org/conversation/the-myth-of-ai#26015

What you quote here is in no way in disagreement with what fchollet quoted. I strictly agree with this. No one is denying the need to not create a raving super-intelligent psychopath. The problem with this fear mongering is that it's not based on any fact, namely, implying that AGI (or SMI, whatever) would by default (naturally even!) be hostile or negligently indifferent to biological life. It could very well be that it would naturally tend to be very friendly, or at least benevolently indifferent. Why would such an entity try to destroy humans?

Not only that, the proposal is to get governments, the US government especially—an organization which in total has not been waging war for only a few decades during it's entire existence—in explicit and tight control of this powerful technology. Doesn't quite seem to be the best idea around, frankly.

There are pretty good arguments that the goals of superintelligent AI will not, by default, be aligned with human values unless we specifically work to make them so. The Russell piece I linked makes (a very short version of) this argument, "A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values...". Nick Bostrom and others build this out in much greater detail.

There are lots of reasons to be skeptical of government regulation. It would probably be nicer if industry can self-regulate and allow governments to stick to the positive role of funding research into the technical questions of building provably controllable/friendly decision-theoretic systems. But a lot of people are claiming that the unified consensus of AI experts is "there's no problem here at all", and I think it's important to push back against that.

Right, but why do fear mongers take it for granted that the subset k is likely to be “exterminate, exterminate”? Look, every AGI researcher I have ever read was well aware of this, and it's their primary concern. It's the central question really. Not so much what n and k to set, but how to get such an entity to do anything at all. Why do biological entities do anything? They have motives. Why do they have motives? Because they have biological constraints, and a hardwired self preservation drive. How to get that in a machine made by us rather than a billion years of natural process, that's the hard question. How to make a machine with the equivalent of motives and emotions. Both sides of the problem are well recognized.

In an ideal world I would prefer this research (AGI, and there is a quantum leap to be made between self driving cars and AGI) to stay strictly under the auspices of “pure science”, i.e. academia. No government, no private interests. But that's not going to happen. The next best thing, I think, would be to not let those in power monopolize the research.

> Not so much what n and k to set, but how to get such an entity to do anything at all.

I don't understand what you think the problem here is. Computers are machines that run programs; my laptop doesn't require an evolved self-preservation drive and set of motives to be convinced to boot OSX every morning. If we program a machine to solve an optimization problem, the machine will just do it, assuming the problem is solvable under whatever algorithm we implement. If the optimization problem involves a sufficiently flexible representation of the real world, and if the program's architecture is set up to pipe solutions into some sort of mechanism (even just a text interface) that acts upon the world -- as it would need to be, to interact and learn from experience -- then the machine will act.

What actions it will take depend on its utility function (i.e., on the optimization problem we gave it), but the vast majority of utility functions will lead to actions that are not aligned with human values. The space of utility functions is incredibly vast, an artificial agent would not be subject to the evolutionary constraints that caused humans to mostly fall within a particular very small region of that space, and pretty much all utility functions lead to behavior like "try to avoid being switched off" and "try to acquire more computing resources" since those are helpful instrumental goals for a wide range of tasks.

Sure, we're nowhere near having a "sufficiently powerful optimization algorithm" and a "sufficiently flexible representation of the real world". But these tasks are essentially THE goal of modern AI research. So it's worth considering what will happen if this research agenda succeeds, whether that happens in a decade, a century, or a millenium.

You're sweeping under the rug the question of what is the optimization problem for such a machine in the first place. There is a huge difference between an agent capable of solving a set of optimization problems when told to, and an agent capable of even human level problem seeking and solving. I agree with what you say. If you give the task of solving a particular problem to what is essentially a glorified Watson, and give it the means to freely act to solve that specific problem, the results could quite well be catastrophic. I'm taking as a given that this is understood, that it's taken as a starting assumption. So, the problem comes down essentially to solving the “help humans” problem. Not “help them vacuum the rug”, or “drive them to work”, or even “solve our energy troubles”, “achieve world peace” etc. Roughly, k<n becomes k=n, where n = “do everything humans do, only better”. Operationalizing such n is the problem.

Reply on news.ycombinator.com