Hacker News Reader

Comment by AndrewKemendo | Hacker News Reader

They aren't separable. The thing that makes you empathetic towards your own family, is also the thing that makes you go kill the other tribe.

"Morality" isn't an actual functional measurable thing, and there is no consistency across cultures so that is right out of the window. Culture has as much to do with behaviors as differences in the brain.

The difference between the "ethics/morality" of an un-contacted tribe in Papua New Guinea and those of Finnish royalty are staggering - yet they have effectively the same hardware.

People need to understand that you can't create something more powerful/smarter than you and expect that there is a way to ensure it won't kill you.

View on HN

In terms of hormonal emotions, which would be the most direct sense of value that we have is completely different. Love might feedback into aggression, but they are very different.

This is view is also very naive but it's the breakdown of the "good" and "bad" discussion. Nobody commits war crimes for their intrinsic value. This is akin to the saying, nobody likes to think they're the bad guys. Even, there are no bad values since the absence of value is the worst.

Naivety is not necessarily wrong, it's just the lack of detail which often leads to the wrong results.

I'm not talking about hormonal emotions. I mean calculated decisions like "should we bomb lybia because they are creating nuclear weapons."

That is a question about an (arguably) existential threat, where force is used for (arguably) protective purposes.

The question is, if the AGI thought that a human was threatening it unprovoked (which would probably happen) would it just let itself get "killed" simply because it was a human doing it?

reminds me of this comic

http://www.smbc-comics.com/index.php?id=4141

Zach has a lot of great comics that accurately represent how a real AGI would probably work.

http://www.smbc-comics.com/index.php?id=4122

I don't know what your point is. Yes some humans are assholes. But most of us aren't. Most people don't want to hurt others. Most have the ability to empathize with others. This comes before culture, it's a part of our brains. Although cultural values are important too.

We want an AI programmed with our values. One that wants to serve us, wants humanity to exist, doesn't want humans to suffer, etc. Defining these values exactly is impossible, which is why solving this problem is so difficult.

You don't have to be an asshole to be dangerous though is the point. When you are thinking of hurting someone it's not typically due to malice, hate it's usually because it's a threat and out of self defense. That's a core part of being an animal/living thing, so it would be no different for an AGI.

Defining these values exactly is impossible, which is why solving this problem is so difficult.

I say it's impossible to solve because defining it is impossible. No two groups have the same end goals, so there will be conflict for survival at some point - and that is inherently dangerous to one group.

Even if you programmed an AGI to want humanity to exist, it could determine (probably rightfully) that humanity is the biggest threat to humanity, and just lock us into a pleasure loop as a species. These and many other scenarios have been discussed ad-nausea in which an AGI with human goals ends up giving us what we want and it's not something anyone would ask for, but would give the end result.

Ironically, this is what was described in the first Matrix movie - and if you take it to it's logical extent it makes sense. I'm not even a sci-fi person so it almost pains me to reference sci-fi, but I think it's apt:

Did you know that the first Matrix was designed to be a perfect human world. Where none suffered. Where everyone would be happy. It was a disaster. No one would accept the program. Entire crops were lost. Some believed that we lacked the programming language to describe your perfect world. But I believe that as a species, human beings define their reality through misery and suffering. The perfect world would dream that your primitive cerebrum kept trying to wake up from. Which is why the Matrix was redesigned to this, the peak of your civilization. I say your civilization because as soon as we started thinking for you it really became our civilization which is of course what this is all about. Evolution, Morpheus, evolution, like the dinosaur. Look out that window. You had your time. The future is our world, Morpheus. The future is our time.

Reply on news.ycombinator.com