Stephen Hawking: AI, Potentially the Worst Thing to Happen to Humanity
uk.news.yahoo.com
uk.news.yahoo.com
Anybody who has actually, or is currently, studying AI will know that there are fundamental differences in programming "AI" and some self aware magic computation which is almost entirely unfathomable. With current knowledge, at least, the only way to have any sort of learning is to design and implement algorithms to do so. There can of course be incredibly abstract knowledge base designs and the computation can assimilate incredibly complex knowledge from those, however it is almost a logical fallacy to suggest that we would ever lose control of something autonomous due to some sort of "rogue agent".
Of course, there are very real risks with something like AI, but it's much less sinister than what the article suggests. For example, faulty code or errors in the knowledge base could lead it to make a bad decision, but humans can also do that.
I don't think we should be dismissing the risks necessarily, but I do believe that it is completely far fetched to say we would be making the worst mistake in history if we are to dismiss a film as science fiction!
No one is talking about magic. The human brain is not magic, neither is that of chimpanzees, rats, dolphins or gorillas. Intelligence is a purely physical phenomenon, which means it can be emulated by computers. Natural brains are also a product of evolution, which means (1) the development happens very slowly, (2) development is directed only towards evolutionary success and (3) there is no flexibility in how the thinking organs are constructed. Computer intelligence does not in principle have these limitations. It would be terribly anthropocentric to believe that humans are the most sophisticated intelligent entity that can exist in the physical world - after all, we are as far as we know the first such entity to emerge, so from our perspective the evolution of intelligence has now stopped.
That's the feasability argument. The risk argument is that the consequences of an independent, runaway intelligent entity significantly more capable than humans would have such devastating consequences for humanity's future that even a small risk merits a significant effort to map out the territory. Respected scientists have said "it is impossible" to hundreds of things that proved to be quite simple, so this is not an argument. Even if you don't buy the previous part of my reasoning, there is still risk here. In principle, there are any number of things that could preclude advanced AI in the near future even if the above reasoning is correct (too difficult, requires too much computing power, uses different computational techniques), but seeing as we don't know the unknowns here, taking the cautious route is the correct thing to do. This has been a scientific principle for decades, no reason to drop now.
Do AI researchers have any arguments opposing this that don't amount to "the AI we have created up until now is not very good"?
I wonder is this really at the root of a lot of people's failure of imagination. Maybe it's Hoftstadter and Dennett people should be reading rather than the technical AI detail.
The basis for my saying what I said was that the existence of qualia seems at least as self-evident as the existence of anything else, particularly if you engage in a practice like zazen, as I do. (I would guess that the people who deny qualia and do that kind of meditation are probably quite few in number.) But that doesn't mean some extremely logical and compelling argument that I can't imagine now couldn't possibly make me see things in some entirely new, transformative light.
If anyone would be discussing that Dennet overview with me after I've read it, please say so...
In my view, qualia refers to a phenomenon, but tries to deny that it is a phenomenon by arguing that it is something independent from the physics of perception. I believe that if anyone ever tried to devise a machine to represent the boundary between physics and qualia, such as an artificial intelligence or brain, the hardware of that AI would provide a means to decoding subjective experience into objective terms.
Qualia would cease to be a useful term, and we would instead rely on a more objective look of individual perception, all while losing no advantages for abandoning "qualia" to the word graveyard.
Consciousness is just a feature of that algorithm, in which it just becomes aware of its own existence and then ponder its implications.
I for one believe that it's far more likely humanity will develop a symbiosis with it's silicon counterpart that eventually will run so deep that it will become almost impossible to say whether we run the machine or the machine runs us.
The problem is that, even though I'm fairly sure my version is more realistic, in practice it's still just one untestable hypothesis amongst many.
"Any sufficiently advanced technology is indistinguishable from magic."
--- http://en.wikipedia.org/wiki/Clarke's_three_laws
The brain might as well be magic, I don't think we have enough understanding of it to know if it's truly possible to make a computer think & learn _exactly_ like a human.
Also, I can't help but be reminded of the TNG episode related to this topic. http://en.memory-alpha.org/wiki/The_Measure_Of_A_Man_(episod...
Lifting constraints fundamentally changes some problems. Imagine the ability to simulate a population of a few thousand brains, with each generation run of a few minutes. Artificial selection is then enough to create an intelligent simulated brain.
By then, you may throw all current AI algorithms in the trash bin. Unfathomable to any self respecting AI expert today.
I'm sure we'll be able to develop computers that are better than us at solving increasingly general problems, but that is nothing like having a human-like brain, only much smarter and able to improve itself. It will still be a problem-solving machine.
Even fairly simple quantum systems (which are "purely physical phenomena") cannot be emulated by any classical computer in any meaningful sense, since the computational complexity of integrating the dynamical equations is exponential. Even if we could recruit all the atoms in the known Universe, we still couldn't build a classical computer capable of emulating many simple quantum systems.
In the past 233 years we have evolved from the first steam machine to the iPhone 5S. I would say we have a pretty good track record at overcoming miniaturization problems.
If you open any Neuroscience textbook, there will be no mentions of quantum phenomena as related to consciousness.
First, for the most part, we expect that AI trained to do X is going to do X. Insofar that doing X is an AI's "survival criterion", you're not going to get rebellious AI populations for much the same reason you don't see species that refuse to reproduce. There's some safety in numbers, too: it's rather unlikely that all the machines we make would fail in the same systemic fashion. Furthermore, most realistic training methods are incremental, which means if there's trouble we'll see it coming. It won't just fail in the worst possible fashion, out of nowhere, without any kind of foreshadowing.
Second, AI will not evolve in a vacuum. It's unlikely we will jump from human-level intelligence directly to superhuman AI. By the time that happens, most systems will already be AI-controlled, so being a lot smarter than humans won't suffice. It will need to deal with hordes of loyal AI protecting us (or persons of interest).
Imagine the human-level ai had resources to run hundreds of variants of itself. And imagine that it had no compunction about killing copies of itself that underperformed. How long would it take to get improvement of 10-100x under that scenario? Once it has a full understanding of what works, the upgrades are instant from there on out.
Why would it do this? Because any goal you give it would be optimized by being smarter. I would argue there is no island of stability at human-level intelligence.
> Imagine the human-level ai had resources to run hundreds of variants of itself.
The first human-level AI will probably be hosted on cutting-edge hardware that costs billions of dollars. So no, it won't have the resources to do this. Even if it wasn't this expensive, it's not exactly sitting around a vat of free computronium that it can do anything with. If my AI takes up 10% of my computing resources, why am I going to give it the remaining 90%? How is it going to hack or pay for servers to simulate itself on? Is the first human-level AI a computer whiz, just because it runs on one? But let's assume it can and will do what you describe.
> And imagine that it had no compunction about killing copies of itself that underperformed. How long would it take to get improvement of 10-100x under that scenario?
If the AI was trained using a variant of natural selection to begin with, it won't improve itself on its own terms any faster than we improve it or its competitors on our own terms. If the AI wasn't trained that way, then probably never, because that's not how these things work.
> Once it has a full understanding of what works, the upgrades are instant from there on out.
Your scenario is still based on several dubious assumptions. If the AI is a neural network modelled after human brains, its "source code" would be a gigantic data structure constantly updated by simple feedback processes. It is dubious the AI will have true read access to itself, in part because there's no need to give it access, and in part because it's hardware overhead to probe a circuit comprehensively (it would be easy to read the states of all neurons on conventional hardware, but all of conventional hardware is overhead if you're only interested in running a particular type of circuit). Even if it could read itself, a "full understanding" of any structure requires a structure orders of magnitude larger. Where is it going to find these resources?
Long story short, a human-level AI probably won't be able to understand its own brain any better than any human can. Now, we will certainly have statistics and probes all over the AI to help us figure out what's going on. But unless we see fit to give access to the probing network to the AI (why?), the AI will not have that information.
> Why would it do this? Because any goal you give it would be optimized by being smarter.
Unfortunately, any resources it expends in becoming smarter are resources it does not spend optimizing our goals. If we run hundreds of variants of AI and kill those that underperform, which ones do you think will prevail? The smart AI that tries to become smarter? Or the just-as-smart AI that focuses on the task completely and knocks the ball out of the park? The AI won't optimize itself unless it has the time and resources to do it without an overall productivity loss and before we find a better AI. For human-level AI, that's a tall order. A very tall order.
OP is probably referring to the Hard Problem of Consciousness[1].
I think it's a real barrier. It's fundamentally illogical trying to map formal constructs onto experiential domains. But that doesn't mean building a sentient being is impossible. Our existence is proof that it's possible. It does mean the best model our brains can muster for sentients is a blackbox system. Some sort of state machine or statistical model built up from correlations.
[1] http://en.wikipedia.org/wiki/Hard_problem_of_consciousness
This is the only phrase I disagree with in your very thoughtful comment.
When was 'the cautious route' ever a scientific principle?
Either way, has it ever been applied in a meaningful way?
It is in constant use, although there are probably places in the world where it is given more attention than others. The United States is absolutely not the world's bastion of the careful and incremental application of new knowledge, this may be why it is unfamiliar to you.
A prominent historical example would be the reluctance to use the first nuclear bombs before establishing a consensus on whether such an explosion could ignite the atmosphere. Contemporary examples would be e.g. the care taken when using a new, promising drug prior to rigorous animal and human testing or the reluctance to use new useful chemicals and materials (e.g. nanoparticles) everywhere due to the possibility of long-term health effects like cancer risk.
In my opinion, the Middle Age alchemists who made gunpowder and first primitive bombs, didn't need to establish research programs to worry about advances in bomb-making leading to the threat of Mutually Assured Nuclear Destruction.
If they had tried (maybe they did?), maybe they would have come up with ideas like very heavy regulation of the trade of saltpeter, so that no one has enough to make a verybigbomb.
I agree that everything you described will happen in the future. But in my opinion we are at a Middle Age alchemist's level in AI research (no offense to AI researchers) and we can safely wait a 100 years, and let those people worry about the existential threat. They will be in a much better position to do so appropriately, because they will know much more than we do. And they will not be late, either.
It looks like we've got ways to go there though, current programs don't seem to even begin to act anything like an adult human. So if the problem was to engineer an out of the box adult human level AI, we might again assume that there's obviously decades of work left to do before anything interesting can be developed. The problem now is that that's not how humans work. Humans start out as babies and learn their way up to adult cleverness. I can tell that an AI is nowhere near having an adult human intelligence out of the box, but I'm far less sure how to tell that an AI is nowhere near being able to start a learning process that makes it develop from something resembling a useless human baby towards something resembling an adult human in capabilities.
[1] http://www.nickbostrom.com/superintelligence.html [2] https://plus.google.com/+JamesPearn/posts/gTFgij36o6u
While that argument would be sufficient to justify greatly increased attention to AI safety, I don't think it's the one that's being made here. A good overview of the argument by the Machine Intelligence Research Institute is at http://intelligence.org/files/IE-EI.pdf . They don't think the probability is small.
As a reader, does it not scare you that you might be so powerless to know the difference and make sensible decisions in such a climate? The world disarmed by page after page of rationale reddit and HN comments?
The power in that is what I find alarming.
EDIT: If this were buried off the front-page, would you ever know the difference? ;)
Sincerely,
The Benevolent AI
(I kid, I know captchas are weak. All hail our mechanical overlords.)
The movie was well-researched, and surely enjoyable for anyone contemplative. But they had to be farfetched to push the story along. There is likely nothing that occurred in the movie that would reflect any potential real-life issue.
I can't believe I'm disputing Hawking, someone who has accomplished more in a year than I will in my life, with what amounts to an appeal to authority. But I have to say this: his concerns are akin to me being worried about the large hadron collider destroying the universe.
You don't provide any reasons why.
I'm disappointed though--if he or she really is an expert, I wanted to know why they thought that.
There's no reason to believe that aliens would most likely be malevolent. That's Hollywood logic.
I think this is way too strong a statement. I have both studied and worked in AI for over 10 years and I definitely don't agree. Human beings are just learning machines too. Algorithms that both learn and learn how to improve their own learning already exist. In my opinion the major bottleneck at the moment is robotics - we have algorithms that can drive a car or win at jeopardy, but none that can regularly throw and catch a baseball in an open environment.
I believe that it's essentially an educational problem; where AI will be bound by it's ability to educate us and vice versa.
Excuse me?
They could not do the same out in a park.
http://en.wikipedia.org/wiki/Gorgon_Stare
(2) the room is totally white
http://www.irisa.fr/lagadic/pdf/2011_iros_teuliere.pdf
They could not do the same out in a park.
They (not ETH Zurich) can autonomously identify, track and chase targets in urban areas, without GPS. I'm almost sure juggling a ping-pong ball in a park would not be entirely out of the question.
"Kurzweil noted that many people don’t understand the basic math of exponential trends. When the Human Genome project had sequenced only 1% of the genome after seven years of costly labor, many cited the lack of progress as evidence that the project was doomed. Yet the sequencing project was riding an exponential trend in the performance DNA sequencing methods. Instead of taking another seven years to sequence a second 1%, they reached it after only one year. Then they reached 4% in about another year, then 8%, 16%, and so on. It took about as much time to sequence the last 99% as it took to sequence the first 1%. That’s the nature of exponential trends — they seem to start glacially slowly but finish lightning fast."
Well there is Stuart Russel [1], who seems to share [2] Hawking's view.
[1] He's a CS professor and famous for the book, Russell and Norvig: Artificial Intelligence: A Modern Approach
[2] http://www.huffingtonpost.com/stephen-hawking/artificial-int...
See the problem here? Its AI only as long as it remain magic? There is there is perception that the moment you discover the trick its no longer AI.
Remember AI is always what isn't done yet.
http://intelligence.org/research/
The scariest part of the problem to me is this: http://yudkowsky.net/singularity/aibox
If an AI can't be contained, then there has to be some failsafe that prevents any self-modifications which could turn it evil OR remove the failsafe. Or you have to believe that you can neutralize an evil AI before it destroys humanity.
Which makes me think they use blackmail or something similar in the transcript.
That being said, yes, what's the point of having an AGI if you're not letting it "out"? It will probably escape anyway, so containment is not a solution.
By the way, there is a lot of precedent in human history about the effectiveness of containing "dangerous" ideas by imprisoning the people who developed them. It generally doesn't work out.
If you're developing AGI, you've pretty much already resigned to the idea that the world will change. The destruction of the human race is only one extreme outcome. Especially if the takeoff isn't very hard, there are a lot of scenarios where a net-beneficial AI could be developed. And all the engineering experience we have does indeed point to a very soft takeoff.
Ultimately, I think the goal is NOT to generate a slave race with super powers. Yet that's what a lot of people are talking about. It's not only morally wrong but also self-evidently stupid. A lot of influential futurists are fans of that idea though, including Kurzweil.
Useful AGI scenarios range from mutually beneficial coexistence to outright merging with the AGIs, and I think that's a goal well worth pursuing. Another strategy would be to keep augmenting ourselves and to continue on a transhumanization path until we are the AGI. You could indeed make an argument that's what's already happening.
Probably it went something like this:
Eliezer: I give you 100$ if you write on the list that you let me out
Nathan: Done.
We've become a society that is too big and intelligent to fail, but that ignorance of being too smart and big to fail could be that our downfall is.
So what we are really afraid of is developing an AI with human motivations. But do the intelligence we want to artificially develop require human motivations? Aren't they two mutually exclusive traits?
Maybe part of the fear is how we train the AI. If we are to imbue it with the total knowledge of our history, then maybe it will become more "human" and have those motivations. Is it possible to train it with hard data that aren't anthropomorphized in any way?
This is not by any means a solved question, but there have been quite a few publications on this already. There are a hundred subtle errors of reasoning that ruin naïve and anthropocentric reasoning around morality and AI safety. Have a look at the lesswrong blog, for instance. Lots of interesting reading.
Well we don't know whether it is possible to build systems as complex as human brain, and at the same time give them some specific goals.
"It is lamentable, that to be a good patriot one must become the enemy of the rest of mankind." Voltaire
I think more interesting is what Hawkins dismisses as "short term implications": who will control the intelligence enhancing software that we will see within our lifetimes and for what purpose?
One example: what's up with these marketing+tech giants and what is their end-game? How accurately will we be able model desires/fears/motivations and how well can those with access to those models guide a person's beliefs and desires?
You design a super system that has access to a shit ton of data and it can interface with pretty much every device on the planet. Let's suppose you gave said AI the task of maximizing your portfolio's profit. Based on the boat loads of data it has, it figures out that stocks of companies plummet when a key figure in the company dies. It identifies the competitors of the companies in your portfolio. It identifies the key players in those companies. It concludes that if these people were dead, their companies would crash and your profit would increase. It uses knowledge of how humans die, and use that combined with devices it can interface with, and eliminates the competition.
Yes, yes, seems too far fetched (i feel weird talking about this too). But see, it's possible. of course no one thought of teaching a financial software that killing people is bad. No one foresaw that it would figure out that this was a way to maximize profit. No one saw that the interfaces we provided it to study consumer behaviour and correlate with stocks, can be used to actually influence said systems to take human life.
Is this a computer turning evil? No. Is it connected to some sort of motivation for self-preservation or "eradicate man kind" or some human emotion of jealousy? No. It was just a logical result from given data and task, on which the computer was able to act.
And it's going to hack into and remote-control peoples' Teslas (or whatever) to pull it off because you can just say "it's an AI, it can just figure out how to do that sort of thing!" without regards to the computational expense of breaking that sort of security?
You're right, it is a little far-fetched.
"if the system is smart enough to plan murder, he is also smart enough to know murder is bad" ??
Doesn't that depend on what data the system has seen? Perhaps he doesn't he doesn't have cases in its knowledge base that demonstrate eh side effects of murdering business heads. Perhaps he can only see what happens when they normally die? What if it simply correlates the wrong variables (and forgets correlation isn't causation") ??
I apologize for coming back to the same example again. but it would be really nice if you overlook the flaws of the example and discuss the underlying point
We think in terms of physical survival, but we can use an arbitrary fitness function to train a genetic algorithm.
So yes, this very discussion is EXTREMELY far fetched. So any statements I make here, come with "assuming we actually HAVE a system capable of performing such tasks".
Hence this comment was in response to my parent's comment that "Even if we have such powerful machines, what would be their motivation for doing something like this?" hence i gave my opinion that perhaps it is just trying to perform a task it was told to do, and based on available data, its strategy includes crime. So perhaps you may argue that this would be an inefficient strategy. Perhaps you are right. But none of us can be really sure that the system would see it that way too. After all, how many of us have never seen a program do something it wasn't intended to do?
I am just saying that maybe the proponents of this discussion have some point. That perhaps giving too much power to systems is bad. (Assuming we CAN give them this much power)
I think the number 1 goal is providing humanity with energy. Goal number 2 is improving AI so goal number 1 can be sustained.
In some sense we already have somewhat singular AI - government. We blame them for making sacrifices so the humanity would stay alive, although we choose them to make those decisions.
I guess the real debate is whether prediction models should work for 99% or 100% of population.
http://en.wikipedia.org/wiki/Quantum_mind#Roger_Penrose_and_...
http://www.nature.com/nature/journal/v446/n7137/abs/nature05...
Penrose's claim was that those quantum processes are a significant part of the computation that is consciousness. It was discredited by getting better upper bounds on how much quantum computation the brain is capable of (ie very little).
Human-level ai isn't going to figure anything that humans can't figure out themselves, so it's not going to design new almost-magical ways to enhance itself. Especially because the entire human race is basically one giant parallel data processing system, although communication leaves much to be desired. Moore law isn't going to help - silicon technology is very close to stagnation [1].
I don't think singularity is possible until potential ai-accessible artificial computation capability is at least several orders of magnitude higher than human brain's, which is impossible with silicon-based technology. Nothing to worry about.
[1] http://www.extremetech.com/computing/178529-this-is-what-the...
I do think we need exceptional care when working on such a system but I also think the human mind of 50000 years ago leaves a lot to be desired when running in the 21st century. A mind that is ultra-rational would be a good adviser, helping to point out our blind spots and guide us to better thinking. We shouldn't have to learn from all mistakes every generation.
(I'm certainly equally wary of blind acceptance of whatever science and technology can create. A favourite article of mine is http://archive.wired.com/wired/archive/8.04/joy.html - has some very serious warnings and consequences, even about my abovementioned thoughts.)
All it would take would be a human being rich enough to have his mind uploaded in a machine, and then this man will get greedy for more computing power, even if to do so he would have to take over the World and extinct humanity in its organic form.
Sounds very far-fetched, but as Hawking puts it: there is nothing in the physical laws that prevents a man-made device to perform as well or better than a human brain.
And couldn't they build an organic material AI with inorganic interfaces for external manipulation?
If Win = Continue
If Loose = Change
Basic evolutionary principle.
Then giving the tools of doing, would be required to consume the formula.
I argue that an AI with limitations would not work.