Hacker News Reader

The issue isn't "evil AI". Very few people in the transhumanist/AI movement are concerned about evil AIs. The issue is that unless we are very careful to engineer an AI that cares about human values it's going to be completely indifferent to human values. Indifference is the enemy, evil is not.

To illustrate, consider the African Elephant. After it reaches 40 to 60 its teeth fall out and it slowly dies in agony from starvation. This is not because nature is evil or because natural selection is evil. This sort of thing just happens because nature is completely indifferent to human values. An AI, just like a process like natural selection, is just not going to care about humans unless we get it right. And since we have only one attempt to get it right, the stakes are so absurdly high.

Anyway, the guys at LessWrong have put a great deal of thought into these issues. If you're interested in this sort of thing, check out the sequences[1]. It's a few million words, but it has a terrific ratio of text to insight.

[1] http://commonsenseatheism.com/?p=12774

Sure, "Evil AI" was just a quick and sloppy way to describe an AI acting detrimental to humans, I didn't actually mean evil. Sorry, my mistake.

The thing is - in order for a true AI to act independently it has to have a purpose. Humans act the way we do because we want to have some sort of positive experience and want to avoid negative experiences. If we couldn't experience anything either positive or negative we wouldn't have a reason or motivation to do anything, we would just be. At that point we might as well be static dead objects. For AIs to act intelligently & independently and not just as algorithms solving a single task they need to have some sort of purpose/goal to reach. An independent AI can't be indifferent - it needs a basis for taking decisions.

I don't see how the purpose we give the AI could be detrimental for humans without severe negligence. In addition - for an AI to be useful for humans it needs to understand humans. We obviously create AIs to serve us and in order to serve us independently it without needing manual input of tasks (which would just make it an advanced computer) it needs to understand us.

Let's say that we tell the AI to eliminate malaria.

So it incinerates the biosphere. Now we don't have any malaria, but we also don't have any humans.

The AI would have done exactly what we asked it to do, but not what we wanted to do. For any reasonable request, you need to specify a ridiculous amount of background information as to what is and isn't acceptable. Probably any simple list you create will be missing something, and we'll be miserable/unhappy as a result of it's exclusion.

I didn't actually mean evil.

I think "evil" is exactly the right word. An entity with goals which are incompatible with your goals (in the sense that both of them cannot be reached) can be described as literally evil. That's exactly what "evil" means.

I agree with what you're saying, but since the nature of your objection is semantic...it depends on what one means by evil. I'd say killing me to make paperclips from my atoms is evil.

The concern is over how to eliminate a certain class of bugs from our software as it becomes increasingly complex to the point that no human understands it. If you ask your robot to get you a glass of milk, and it finds you are out, you don't want it to rob the local grocery store, killing anyone who tries to stop it. Fixing such bugs by enumerating things not to do isn't viable -- we have to get machines that share our values or stick to Oracle type machines that just don't care about the physical world.

That's the thing - I don't agree with your premise that humans and robots/AI would be as seperate as you frame it. "Vanilla" humans may not understand the AI, but the people working with the stuff would surely be vastly enhanched humans, cyborgs.

I agree that we have to create AIs that share our values. However, I don't understand how/why we would/could not. We obviously create AIs to serve us and in order to serve us independently it without needing manual input of tasks (which would just make it an advanced computer) it needs to understand us.

I simply don't understand how the default AI would be detrimental to humans, what purpose would such an AI serve and why would we create it?

The mind design space is HUGE. (http://lesswrong.com/lw/rm/the_design_space_of_mindsingenera...)

The "default AI" is a program that we build. That's all we know. Most programs that we build do not properly represent human values. If they don't properly take those into account, then we lose things we care about. The AI that "optimizes our supply chain for paper clips" will, in the limiting case, consume humans and the environment and the earth and the sun in order to produce as many paperclips as possible and distribute them as widely as possible. A "default AI" will not care about its survival or the survival of its creators.

>I've already started to save money for body/brain updates and longevity treatments. I'll gladly admit that it's more important to me than saving for a house/car or my pension. I'll rather live in a crappy apartment/with a crappy car for the next decades and afford to live 150-200 years (when immortality surely will be an option for the rich, which I plan to be) than to spend all my money on a house/car and die a natural death at 80-100 years old.

That is IMHO a bad decision. I was also thinking about this way of life for some time. The big issue here is that you are betting with your wealth life in ages (house, pension, ...) against some possibly available life extension tool.

From my point of view the technology growing rate slightly decreases in the last decade. There is already 2012 outside but we still do not have flying cars (as has been thought twenty years ago), we were not able to beat HIV, we still do not understand how the human brain works, etc etc etc.

So, my bet is that this life extension medicine will exists in a usable and working way maybe in 50 years. At this time your are already 71 or maybe even older. Do you really want to have a long life at this age??? No, I bet not. For us (I am 28) it is already too late now. And I do not want to live 200 years at an age of 80 or so. I would be old crappy man, and will be psychologically labile since there would be so much young people around me living a much longer "young" life as I did. I don't want imagine how this feels like.

> I don't see a logical reason why artificial emotional "beings" would favor the future of purely technological "beings" over humans/cyborgs - they/it doesn't have the same evolutionary drive to advance it's own species as natural life, including humans, have.

Wouldn't that depend on how these "artificial beings" are designed in the first place? They might not necessarily favor a robot-dominated world, but they're not necessarily going to coexist peacefully with human beings, either. Whether they serve everyone's interests or whether they'll need to have their rogue asses kicked by Will Smith ultimately depends on what kind of intelligence and emotions are programmed into them. Even if they are expected to learn on their own, the design of the learning algorithm (as well as the stimuli they're initially exposed to) will have a significant impact on the outcome.

In particular, human philosophical conjectures will inevitably make their way into the design of artificial beings. From abstract concepts in metaphysics and epistemology to the most practical parts of value theory, human philosophy pervades every "intuitive" assumption that we make on a day-to-day basis. But I have yet to see a philosophical position that would be safe if extrapolated to its logical conclusions by hyper-intelligent beings. So there is definitely cause for concern, in addition to well-known issues of inequality.

"Evil" is a label for a particular subset of human behaviors, which are a subset of all possible minds. Thus it is said:

> The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.

The main point of the Friendly AI movement is that the vast majority of possible (stable) minds will not work towards a universe that humans would enjoy living in. An AI that values what we value is a stupendously narrow target - particularly since we can't precisely define our values yet.

If you're interested in this stuff, there's a lot more on LessWrong. Here's a teaser: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

"Evil" is a label for a particular subset of human behaviors, which are a subset of all possible minds.

I think it's far more useful to define "evil" to be about goals which are fundamentally incompatible with your own. Not only does this cover every human example, but it matches the intuitive feel for what "evil" means even if the entity in question is not human, and leaves systems which have no internal representation of goals as morally neutral, like hurricanes, no matter how dangerous or damaging.

> Research has shown that AI needs emotions in order to be truly useful or else it won't have a way to decide what's important or right, leaving it decisionless like people with a damaged amygdala.

Research has shown what, exactly?