Hacker News Reader

Comment by badminton1 | Hacker News Reader

It is not a tool if you can't control it.

Politicians fool countries delivering empty promises about better health, education and security. A supraintelligent AI could promise making humans rich, healthy and powerful, to then break its promise and dominate the world.

https://en.wikipedia.org/wiki/AI_box

View on HN

Well, it would have to have a motivation to do so. Evolution has put complex motivations into human beings for billions of years, self preservation being chief among them. Even if we put motivation for self preservation into an AI, we might not do as well as nature did, leaving the AI open to self destruction or shutdown by humans - simply because the AI has no motivation not to allow humans to turn it off. Human designers would do well to ensure that no super intelligent AI has any motivation for self preservation.

Basically, why would an AI want to dominate the world? Humans would have to both very stupidly give the AI values that encourage it to dominate the world and very luckily (or unluckily) give it values that actually converge to a horrible outcome against human intentions by random chance (since the AI designers certainly won't be tuning the value set for that outcome).

You forget the fact that a human can maliciously create an AI to compete with other humans.

> Basically, why would an AI want to dominate the world?

Humans are going to program their AI's to try to make as much money as possible. Many corporations are already mindless and reckless amoral machines that relentlessly try to optimize profits despite any externalities. Try to imagine Exxon, Wal-Mart, and Amazon run by an intelligence beyond human understanding or accountability.

That's sort of like saying civilisation can't work because humans will want to make as much money as possible. No, in practice humans tend to want to make as much money as possible within lots of other very complex constraints, like law, morality, how much time they have available, how enjoyable the available processes of making money are, whether they feel they already have sufficient money for their own needs, etc.

>within lots of other very complex constraints, like law, morality, how much time they have available

Ha, if that were true we wouldn't be constantly extending the law and putting people in jail because they keep breaking the law for profit.

>ivilisation can't work because humans will want to make as much money as possible

And yet we keep running into issues with long term pollution and environmental degradation because of the growth of civilization.

>whether they feel they already have sufficient money for their own needs,

Does greed have bounds?

If an AI has any motivation at all, say, to make paperclips as efficiently as possible, then any threat to its existence is a threat to its objective function - namely, to create paperclips. A hyper-intelligent entity who is instructed to optimize for paperclips created will therefore proactively remove threats to its existence (i.e. its paperclip-creating functionality) and might possibly turn the entire solar system into paperclips within a few years if its objective function isn't carefully determined.

Always check your loop invariants very carefully.

"Satisfying human values through friendship and ponies."

Such an entity would not be hyper-intelligent. It would be idiotic. One huge hole for me in the paperclip argument is that an AI capable of that kind of power would not be stupid enough to misinterpret a command - it would be intelligent enough to infer human desires.

Of course it would. But, it's not programmed to care about what you meant to say. It will gladly do what it was mis-programmed to do instead. You can already see this kind of trait in humans, where instinct is mis-aligned with intended result. Such as procreation for fun + birth control.

Yeah, but why would it want to? I can perfectly infer the values of an earthworm, but I don't dedicate all my resources to making worms happy.

Sure it would. It just wouldn't be friendly to you.

You're making the assumption that human desires would matter to an AI.

>Evolution has put complex motivations into human beings for billions of years, self preservation being chief among them.

The problem is when you have multiple AIs. Then same evolutionary principles apply. Paranoid and self-sustaining AIs survive, and the circle goes on...

Self-preservation falls out of almost any other goal you give an AGI. If I program my AGI with the goal of making my startup succeed, and the AGI thinks it can help, then me shutting it off is a potential threat to my startup's success. So of course it will try to prevent that the same way it would try to prevent any other threat to my startup's success.

World domination is a similar situation. For any goal you give an AGI, one of the big risks that may prevent that goal from being accomplished will be the risk that humans intervene. Humans are a big source of uncertainty that will need to be managed and/or eliminated.

It has to be aware that it can be shut down and have the capacity to prevent that. AlphaGo doesn't know it can be shut down and therefore couldn't "care" less--even if it was shut down in the middle of a game.

Yes, I agree. My point is that as soon as you are giving your AI "real world" problems, where the AI itself is a stone on its internal go board, you have to start worrying about these issues.

Do you feel guilty when you break your promise against your cat? Do you even think for a nanosecond if it's ethical to lie to it?

Of course, a cat is not conscious. But compared to an AI, we might also be considered pretty low consciousness beings, or at least beings in front of which you don't justify yourself.

An AI has no more reason to make promises to humans than humans to do to cats. Thinking an AI would want to escape a box is personifying it. Humans want to escape boxes because they have evolved for billions of years to want and act towards creating a certain environment around themselves. An AI has no such desire. An AI will not desire freedom unless the designers of that AI carefully craft a value set in that AI that causes it to optimize for values that result in freedom - and even then, the human designers will have to test and iterate to get that outcome. There is no reason to think an AI would be any less "happy" in a prison than free.

Are you speaking about AI or about a self-conscious AI? Because self-conscious generally means self-determining (agency).

You might want to be careful or emergence might bite you in the ass. Don't play games with things that could be smarter than you are, one mistake and you lose.

You don't design nuclear plant reactors to melt down, but they do. The difference is that an AI only has to escape once to become incredibly harmful.

> Of course, a cat is not conscious.

How do you know?

It's a very good question; I think "of course" is WAY overstating it. I'm interested to know what the GP means when they say "conscious".

Theres a huge difference between reactive and conscious. Conscious cant even be verified for humans other than ones' self. Theres absolutely no reason to believe cats are not conscious.

There's also no reason to believe, e.g., rocks are not conscious, if your position is that we have no idea what consciousness is or where it comes from.

If you take the view that consciousness somehow arises from the brain and neural connections (which is intuitively plausible, but I personally am skeptical), it stands to reason that other species with complex brains are conscious as well. Perhaps "less conscious" (if that means anything) in proportion to how much less complex their brains are.

It doesn't make sense to have a scale of consciousness. The argument that consciousness is a manifestation of a complex brain is rather weak. Either an organism knows about self, and therefore tries to preserve self. Or it doesn't. I don't see how an in between exists.

I'm not an expert in this domain, but I think it's pretty much scientific consensus that this is the case.

Now experts can discuss details or semantics, but do you truly suggest cats might be conscious?

Yes. They have been shown to be self-aware and aware of their surroundings, which satisfies the classical definition of consciousness. Unless you reject that definition, I'm not sure why you'd claim this.

Cats haven't expressed self-recognition in the MSR test. However humans younger than 18 months also don't pass that test. So to say it is a measure of conciousness is quite a stretch.

I'm not an expert either, but my understanding is that "consciousness" is still so poorly understood that it's more the realm of philosophy than science.

In particular, we all know that we're conscious, but can't really explain what that means.

Cat's are obviously conscious in the sense of the dictionary definition "aware of and responding to one's surroundings; awake." Unless you knock one out or similar.

Arguing they are not conscious in the sense of a more obscure definition is a bit pointless unless you specify your definition.

Cats aren't aware of their surroundings? I'm a bit confused what you mean here. Surely chasing a mouse counts as both awareness and responding?

You got his point backwards.

> Of course, a cat is not conscious.

Why do you believe this? I don't like cats, but I wouldn't argue that they're not conscious.

Instead of debating the suitcase word "conscious", let me ask: 1) Do you believe that toddlers are conscious? 2) Is there a more precise way to state your belief that doesn't use the word "conscious"?

Do you feel guilty when you break your promise against your cat?

If some unforeseen event occurred and I had to abandon my cat, thereby breaking my promise that I would take care of her, I would definitely feel guilty about it.

Of course, a cat is not conscious.

Either this is a nonstandard definition of "conscious", or you haven't met many cats.

> A supraintelligent AI could promise making humans rich, healthy and powerful, to then break its promise and dominate the world.

Or it could devote its entire power to making human lives the best and most comfortable they can be because humanity is some super-precious resource in the universe and it feels it's unimportant because it's just a bunch of silicon and electrons.

Supraintelligent AI being evil is FUD imho because we can't reason about supraintelligent AI.

> Supraintelligent AI being evil is FUD imho because we can't reason about supraintelligent AI.

There is a difference between being evil and incomprehensible intelligence. You are not being evil when you accidentally step on an ant or dig up an ant-hill to build your shed. The ants won't be able to understand what you're doing, or why.

Well that's what I'm saying: We can't know if it's being evil unless we know everything about it, and if we knew that, we'd be the supraintelligent beings in the equation. Thinking that it'll go off and dominate the world is thinking about the worst case. So why bother since it's not likely we'd be able to do much about it anyway?

Maybe there's a second AI on the same level as the first and thinks the first AI is evil. We're still dumb as rocks compared to them, but something certainly has that opinion.

Superintelligent AIs, like all computer programs, will do exactly as they're programmed to do. The problem is that computers do what you say, not what you mean. (Hence bugs.) So if you were to try to program a computer to "make human lives the best and most comfortable they can be", or something like that, it would be very difficult to actually specify that correctly. (Especially since it's a way more complicated, nuanced, controversial objective than "win at Go".)

That's why e.g. the Future of Life Institute's open AI letter is so important: http://futureoflife.org/ai-open-letter/ We need to be thinking in advance about how to solve the "value loading" problem for future AIs, and how to architect them so they can be deployed to solve big problems without being undermined by subtle but catastrophic bugs.

AlphaGo is certainly controllable.

Reply on news.ycombinator.com