Hacker News Reader

Comment by Moshe_Silnorin | Hacker News Reader

Nobody is afraid of today's AI algorithms. But if we make machines that are smarter than us and have desires, they will influence the future to achieve their desires. If these desires conflict with our own, things will not end well for the dumber party.

As we really have no idea what we, collectively, think of as a moral terminal goal, and less so how to formalize this, there is no reason to expect the first AIs to have goals that correspond to what we want. If AIs self-replicate in a competitive ecology, what would be selected for would be agents millions of times more intelligent than us who use their intellects only to make more copies of themselves - using all available resources including those we need to survive.

I'd recommend people read Stephen Omohundro's paper on the topic, Basic AI Drives: https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...

View on HN

> if we make machines that are smarter than us and have desires, they will influence the future to achieve their desires. If these desires conflict with our own

Aren't you anthropomorphising them here? Just because something might be intelligent, it doesn't follow that they automatically gain other attributes that we have, such as desires. Why would any AI have anything that could meaningfully be called "desires", except in the sense that it is designed to achieve its purpose, e.g. a self-driving car "desires" to get to its destination?

> there is no reason to expect the first AIs to have goals that correspond to what we want.

There's every reason to expect this. Look at the ways in which we use AI at the moment. It's to achieve our goals. We build self-driving cars to get from A to B. We build optimising query planners to make our databases run faster. We build software that designs antennae so we have better antennae. Nobody wants to build software that doesn't do what they want it to. You know what you call intelligent software that has a goal that doesn't correspond to what we want? Broken and useless. Who wants to build broken and useless software?

Making something intelligent does not automatically give it desires. But as you said, there is every reason to expect that we would build AI to achieve our goals, which means making it (a) desire to achieve a goal we share and (b) intelligent enough to succeed. (This is not anthropomorphizing AI; there's no need to make it feel desire the way we do, it just needs to tend to reach a state where the goal is achieved.)

The danger comes from our inability to accurately define our own desires. There is no doWhatIMean() function. A programmed goal can subtly differ from the one we want so that it passes the obvious test cases but breaks catastrophically in an unusual real-world situation. There's an anecdote[1] about a neural network trained to detect camouflaged tanks (I've also heard of it detecting Soviet versus NATO tanks); it ends up learning to detect objects with shadows, because the tanks were photographed on a sunny day and the non-tanks on an overcast day. Now imagine if the program had been prematurely instructed to attack the camouflaged tanks. You can call this "broken and useless software," but by the time we realize how it's broken it could be too late.

[1]: http://www.j-paine.org/dobbs/neural_net_urban_legends.html

> But if we make machines that are smarter than us and have desires

It is a hypothesis that human desires - the conscious processes involved in thinking "I want such-and-such" - influence human behavior. Consciousness is not whatsoever a solved problem, and it may be that we are just complex, deterministic automatons whose consciousness and desires are caused by our behavior (like the steam produced by an engine), rather than causing it.

I'm not necessarily in this camp, but I bring this up to point out that there is an awful lot of common prejudice bundled up in what we consider our own intelligence/consciousness to be, and we project these biases onto what we imagine AI will be. We almost certainly misunderstand our own consciousness profoundly in this way, and at any rate there is no definitive scientific (or philosophical) answer to "what is self awareness" or "what role do desires play in human behavior" (again, the desire could be the steam from the engine).

> and it may be that we are just complex, deterministic automatons whose consciousness and desires are caused by our behavior (like the steam produced by an engine), rather than causing it.

Then explain that we are talking about it.

I'm not sure I understand your point. Computers communicate with one another constantly. There certainly does seem to be something different about us, but I've yet to see any scientific or philosophical proof that our self-awareness is necessarily involved with our intelligence or behavior. There's a good deal of philosophical literature around this idea ( https://en.wikipedia.org/wiki/Philosophical_zombie ), and virtually no scientific explanation of consciousness.

I grant of course that this may not be a terribly useful way for human beings to view themselves, and it is quite contrary to popular belief, but for all that it could still be true.

Just as an example of what you're saying: a few posts up the index, there was an article on intelligence and language in dolphins.

Dolphins are practically our cousins --- they're social mammals living in much the same world we do, and a lot of their behaviour maps directly onto ours. But they're so alien to us that we're still arguing about whether they are intelligent, low alone how much.

If we can't even communicate effectively with them, what chance have we with a machine intelligence with which we have nothing whatsoever in common, not even time and space?

> Nobody is afraid of today's AI algorithms.

I'm more afraid of who will have control over this new form of infrastructural capital, and how they will drive the rest of humanity into poverty.

Everybody can stand on the shoulders of giants, but there's only a few who can monetize it.

>As we really have no idea what we, collectively, think of as a moral terminal goal, and less so how to formalize this, there is no reason to expect the first AIs to have goals that correspond to what we want.

The first part of this sentence is true and interesting, but it makes the second half not very scary. Essentially, since we don't know what if anything, we want morally, what is the danger in letting AI's do whatever? Don't you think if they did do something we objected to, we could respond at that point?

It's like law isn't it - lots of areas of humanity aren't governed by law at all. You can watch someone prepare themselves a godawful sandwich with things nobody would put on the same plate, or write a meandering book that is a horrific read, or wear crocks with a suit, and you can't sue them for any of this, there just aren't any laws at all on these subjects. But they're not really problems on a social level either - laws are created when things become a problem.

So, likewise, where is the huge social problem with AI's? Nowhere. So, why worry? surely it's not like any horrific AI can happen in 8 hours, so that the entire rest of humanity (much of which is completely segregated from any one network, etc) can't possibly respond. Today, AI has no autonomy or control over anything, whereas 7 billion people are organized in societies and oversee millions of dumb computer systems that do whatever people instruct - this is not going to suddenly turn around in a matter of hours, and since, as you state, we have no definitive terminal preferences, I don't see the danger in letting AI control things we don't care about.

I suppose this could be different if AI controlled a lot already, but it's just not the case.

> The first part of this sentence is true and interesting, but it makes the second half not very scary. Essentially, since we don't know what if anything, we want morally, what is the danger in letting AI's do whatever? Don't you think if they did do something we objected to, we could respond at that point?

The problem is that with powerful enough AI, we won't be able to effectively respond. With such a system it is vital for it to have its goals almost perfectly aligned with ours - otherwise, any conflict of even the tiniest of values may mean anything from our value disappearing from the universe to us disappearing from the universe. A lot, if not most of the AI-related problems are in fact corner cases of value alignment issue.

Or, as some people say, any powerful AI that is not explicitly designed to be Friendly (i.e. have values strongly aligned with ours) will bring doom to humans.

Reply on news.ycombinator.com