My response is here: https://news.ycombinator.com/item?id=11295675
1,339 karma · joined June 12, 2010
A correct implementation of a list sorting algorithm does not need to be separately told how to sort every individual list. Similarly, a correctly implemented general reasoning algorithm does not need to be given special instructions in order to reason about humans & human society.
The problem comes when a correctly implemented general reasoning algorithm gets paired with an incorrect specification of what human goals are. And because a correct specification of human goals is extremely hard, incorrect specifications are the default.
>Intuitively we can understand this by thinking about employees. One does exactly what he is told, but only what he is told, and then comes back for more instructions. Another can be given a goal, and then goes off and finds his own ways to accomplish that goal. Which one is more effective? Which one is more likely to compete for his manager's job some day?
The third possibility is that of an employee who goes off and finds their own way, but instead of accomplishing the goal directly, they think of a way to make their manager think the goal is accomplished while privately collecting rewards for themself. In other words, a sociopath employee whose values are different from their manager's.
By default, an AGI is going to be like that sociopath employee: unless we're extremely careful to program it in detail with the right values, its values will be some bastardized version of the values its creators intend. It will sociopathically work towards the values it was programmed with while giving the appearance of being cooperative and obedient (because that is the most pragmatic approach to achieving its true values).
Most humans are not sociopaths, and we have a shared evolutionary history, with a great deal of shared values, shared cultural context, and the desire to genuinely be good to one another. Programming a computer from scratch to possess these attributes is not easy.
If I order someone to prove whether P is equal to NP, and a day later they come back to me with a valid proof, solving a decades-long major open problem in computer science, I would call that person a genius.
>EX: Maximizing the number of people that say Zulu on black Friday without anyone noticing that something odd is going on.
Computers do what you say, not what you mean, so an AGI's goal would likely be some bastardized version of the intentions of the person who programmed it. Similar to how if you write a 10K line program without testing it, then run it for the first time, it will almost certainly not do what you intended it to do, but rather some bastardized version of what you intended it to do (because there will be bugs to work out).
This is a purely semantic distinction. Thought experiment: Let's say I modify your brain the minimum amount necessary to make it so you are incapable of modifying your goals. (Given the existence of extremely stubborn people, this is not much of a stretch.) Then I upload your brain in to computer, give you a high speed internet connection, and speed up your brain so you do a year of subjective thinking over the course of every minute. At this point you are going to be able to quit a lot of intelligent-seeming work towards achieving whatever your goals are, despite the fact that you're incapable of modifying them.
>If a paperclip AI is so dedicated to the order to produce paperclips, why wouldn't it be just as dedicated to any other order? Like "don't throw me in that incinerator!"
The paperclipper scenario is meant to indicate that even a goal which seems benign could have extremely bad implications if pursued by a superintelligence.
People concerned with AI risk typically argue that of the universe of possible goals that could be given to an AI, the vast majority of goals in that universe are functionally equivalent to papperclipping. For example, an AI could be programmed to maximize the number of happy people, but without a sufficiently precise specification of what "happy people" means, this could result in something like manufacturing lots of tiny smiley faces. An AI given that order could avoid throwing you in an incinerator and instead throw you in to the thing that's closest to being an incinerator without technically qualifying as an incinerator. Etc.
http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/
Let me put it another way: Humans are a result of evolution. We know that evolution created us to have as many descendants as possible. But most of us don't care, and we use technologies like condoms and birth control to cut down on the number of descendants we have. Adding more intelligence to humans helps us understand evolution in greater detail, but it does nothing to change our actual goals.
It's relatively easy to formally describe what it means for a list to be sorted, and prove that a particular algorithm always sorts a list correctly. But it's next to impossible to formally describe what it means to be nice to humans, and proving the correctness of an algorithm that did this is also extremely difficult.
These considerations start to look really important if we're talking about an AI that's (a) significantly smarter than humans and (b) has some degree of autonomy (can creatively work to achieve goals, can modify its own code, has access to the Internet). And as soon as the knowledge of how to achieve (a) is widely available, some idiot will inevitably try adding (b).
Note: Elon Musk and Sam Altman apparently think spreading (a) to everyone is a good way to mitigate the problem I describe. This doesn't make sense to me. You can read my objections in detail here: https://news.ycombinator.com/item?id=10721621 There's another critique of their approach here: http://slatestarcodex.com/2015/12/17/should-ai-be-open/
If you're interested to learn more, here's a good essay series on the topic of AI: http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...
But the problem is that computers do what you say, not what you mean. If I write a function called be_nice_to_people(), the fact that I gave my function that name does nothing to affect the implementation. Instead my computers behavior will depend on the specific details of the implementation. And since being nice to people is a behavior that's extremely hard to precisely specify, by default creating an AI that's smart enough to replace humans is likely to result in a bad outcome.
Recommended book: http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
World domination is a similar situation. For any goal you give an AGI, one of the big risks that may prevent that goal from being accomplished will be the risk that humans intervene. Humans are a big source of uncertainty that will need to be managed and/or eliminated.
That's why e.g. the Future of Life Institute's open AI letter is so important: http://futureoflife.org/ai-open-letter/ We need to be thinking in advance about how to solve the "value loading" problem for future AIs, and how to architect them so they can be deployed to solve big problems without being undermined by subtle but catastrophic bugs.
http://www.weforum.org/agenda/2016/03/have-we-hit-a-major-ar...
The line about programmers fearing automation only once it affects them was actually an attempt at a joke :P
The argument I'm trying to make is a simple inductive argument. Once something gets automated, it rarely to never gets un-automated. More and more things are getting automated/solved, including things people said would never be automated/solved. What's to prevent programming, including AI programming, from eventually being affected by this trend?
The argument I laid out is not meant to make a point about wait times, only feasibility. It's clear people aren't good at predicting wait times--again, Go wasn't scheduled to be won by computers for another 10+ years.
Well here's my counter: mistakes either make sense to make or they don't. If they make sense to make, they aren't mistakes, and AIs will make the "mistakes" (e.g. inserting random behavior every so often just to see what happens--it's easy to program an AI to do this). If they don't make sense to make, making them will not be an advantage for humans.
At best you're saying that we'll hold humans of the future to a lower bar, which does not sound like much of an advantage.
Probably not soon. Worth noting that the Go victory happened a decade or two before it was predicted to though.
(To clarify, I was not trying to establish that any intelligence explosion is on the immediate horizon. Rather, I was trying to establish that it's a pretty sensible concept when you think about it, and has a solid chance of happening at some point.)
>Just like other people in other jobs, we sometimes come up with more efficient ways of doing things than our bosses thought possible, and then we have some extra free time to do what we like.
Yes, I'm a programmer (UC Berkeley computer science)... I know.
But don't listen to me. Listen to Stuart Russell: https://www.cs.berkeley.edu/~russell/research/future/
(I'm convinced that programmers will take the AI-automating-all-the-jobs idea more seriously once it's their own jobs that are on the line)
There are a few ways I can think of to object to this line of reasoning:
1) You could argue that programming will be a job that never gets automated away. But this seems unlikely--previous intellectual tasks (Chess, Jeopardy, Go) were thought to be "essentially human", and once they were solved by computers, got redefined as not "essentially human" (and therefore not AI). My opinion: In the same way we originally thought tool use made humans unique, then realized that other animals use tools too, we'll eventually learn that there's no fundamental uniqueness to human cognition. It'll be matter & algorithms all the way down. Of course, the algorithms could be really darn complicated. But the fact the Deepmind team won at both Go and Atari using the same approach suggests the existence of important general-purpose algorithms that are within the reach of human software engineers.
2) You could argue that programming will be automated away but in a sense that isn't meaningful (e.g. you need an expensive server farm to replace a relatively cheap human programmer). This is certainly possible. But in the same way calculators do arithmetic much faster & better than humans do, there's the possibility that automated computer programmers will program much faster & better than humans. (Honestly humans suck so hard at programming https://twitter.com/bramcohen/status/51714087842877440 that I think this one will happen.) And all the jobs that we've automated 'til now have been automated in this "meaningful" sense.
Neither of these objections hold much water IMO, which is why I take the intelligence explosion scenario described by Oxford philosopher Nick Bostrom seriously: http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
The fact that a claim bears superficial resemblance to one made by wackos might be a reason for you to believe that it's most likely not worth investigating, but as soon as you spend more than a few minutes thinking about a claim, the weight of superficial resemblances is going to be overwhelmed by other data. If I have been talking to someone for three hours, and my major data point for inferring what they are like as a person is what clothes they are wearing, I am doing conversation wrong.
"Nuclear weapons are fundamentally different from every other doomsday scenario I cited because they were designed explicitly and solely as weapons whose purpose was to wreak massive destruction at a huge cost to human life. The potential to destroy humanity defined nuclear weapons. That is not the case for other technological, demographic and resource-related doomsday scenarios, which are much more tenuous."
Sure, and in the same way nuclear bombs were a weaponization of physics research, there will be some way to weaponize AI research.
"We already have a world awash in civilization. This civilization is, of course, termite colonies. Termite colonies are effectively miniature civilizations. They have specialization of labor, they build structures much larger than any one organism, and they go to war with one another. Yet even the nimblest termite colonies can only eat so much."
"That is to say, no matter how much an organism evolves, we would be looking at something that is still hard-limited by its environment, even if it can compete with a termite colony's worth of construction ability. The danger isn't that it spreads through the world and destroys entire ecosystems; the danger is that it finds a supply of raw materials and smashes a termite mound or two while building its own bigger house."
Humans don't respect the social conventions of much less intelligent animals that we've domesticated. If an AI much smarter than humans was created, I don't see strong reasons to believe it would respect our social conventions.
http://futureoflife.org/data/documents/research_priorities.p...
Agreed.
>Nor do I think that the impossibility of predicting the future of AI is, in itself, a reason for undue caution.
Sure.
>Bostrom is performing a sort of Pascalian blackmail by claiming that the slight chance of utter destruction merits a great deal of concern. In fact, he is no different from a long line of doomsday prophets who have evoked fictional and supposedly superior beings, ranging from deities to aliens, in order to control others. The prophet who awakens fear puts himself in a position of power, and that's what Bostrom is doing.
Consider: "Previous books published by authors hailing from Country X contained flaws in their logic; therefore since this book's author came from Country X, this book must also have a logical flaw." It's not a very strong form of argument: you might as well just read the book to see if it has logical flaws. Similarly even if a claim seems superficially similar to the kind of claim made by non-credulous people, that's far from conclusive evidence for it being an invalid claim.
It would be a shame if religious doomsayers have poisoned the well sufficiently that people never listen to anyone who is saying we should be cautious of some future event.
>And you and I both know, most if not all of those threats have turned out to be nonsense -- a waste of time.
Sure, but there have been a few like nuclear weapons that were very much not a waste of time. Again, you really have to take things on a case by case basis.
>His notion of superintelligence is based on a recursively self-improving AI that betters itself infinitely. Most technological advances follow S-curves, moving slow, fast and slow. Bostrom does not seem to grasp that, and cites very little evidence of technological change to back up his predictions about AI. He should be, first and foremost, a technological historian. But he contents himself with baseless speculation.
I can see how this could be a potentially fruitful line of reasoning and I'd encourage you to pursue it further, since the future of AI is an important topic and it deserves more people thinking carefully about it.
However, I don't see how this does much to counter what Bostrom wrote. Let's assume that AI development will follow an S-curve. This by itself doesn't give us an idea of where the upper bound is.
Estimates suggest that human neurons fire at a rate of at most 200 times per second (200 hertz). Modern chips run in the gigahertz... so the fundamental activity that chips do is being done at something like a million times the speed that the fundamental activity neurons do is being done. (Humans are able to do a ton of computation because we have lots of neurons and they fire in parallel.)
And there are lots of arguments one could make along similar lines. Human brains are limited by the size of our skulls; server farms don't have the same size limitations. Human brains are a kludge hacked together by evolution; code written for AIs has the potential to be much more elegant. We know that the algorithms the human brain is running are extremely suboptimal for basic tasks we've figured out how to replicate with computers like doing arithmetic and running numerical simulations.
Even within the narrow bounds within which humans do differ, we can see variation from village idiots to Von Neumann. Imagine a mind V' that's as inscrutable to Von Neumann as Von Neumann is to a village idiot, then imagine a mind V'' that's as inscrutable to V' as V' is to Von Neumann, etc. Given that AI makers do not need to respect the narrow bounds within which human brains differ, I don't think it would be surprising if the "upper bound" you're discussing ends up being equivalent to V with a whole bunch of prime symbols after it.
(There's also some DH2 http://paulgraham.com/disagree.html level stuff about the terms Bostrom chooses to use to make his argument... I'm not sure if there's anything to be said about this except that Bostrom's book seems more accessible to me than most academic writing http://stevenpinker.com/why-academics-stink-writing and I'd hate for him to receive flak for that. It's the ideas that matter, not the pomposity with which you communicate them. I also don't understand the implied disregard for anything that seems like science fiction--what's the point in trying to speculate about the future if all the speculation will be ignored when decisions are being made?)
If an Oxford philosophy professor is not enough for you, here's a succinct explanation of the AI safety problem & its importance from Stuart Russell (Berkeley CS professor & coauthor of Artificial Intelligence: A Modern Approach):
Ng's statement is, at best, equivalent to a student who is putting off starting their semester project until finals week. Yes it seems far away, but the future is going to happen.