Comment by robertk | Hacker News Reader

Comment by robertk | Hacker News Reader

"What we don't have to worry about is the AI algorithm running them, because that's speculative. There isn't an AI algorithm that's good enough to do that for the time being."

By the time a risk does hit, it will be far too late to invent the requisite mathematics necessary for machine safety. There are many paths towards dangerous AI futures and many powerful forces that could inadvertently tip the boat the same way we are escaping the implicit "goals" of our genes, so the union of the respective probabilities is very much not speculative and must be taken seriously.

"There's a whole other problem area that has to do with neuroscience, where if we pretend we understand things before we do, we do damage to science, not just because we raise expectations and then fail to meet them repeatedly, but because we confuse generations of young scientists. Just to be absolutely clear, we don't know how most kinds of thoughts are represented in the brain. We're starting to understand a little bit about some narrow things."

Appealing to scientific ignorance is always a bad idea: http://lesswrong.com/lw/kj/no_one_knows_what_science_doesnt_...

Forgive my lack of knowledge on the topic but a question keeps popping into my head when I read comments about the danger to humanity where AI runs amok.

> There are many paths towards dangerous AI futures,

Are there not just as many paths towards protective, helpful <or insert one of many adjectives here> AI futures?

Is there a reason to believe that there will only be one AI in the future and that given a directive to do something, the elimination of humanity will be a logical endgame scenario for it?

Why not many AI entities with different and competing goals? Granted, this opens up a different can of dangerous worms. But still, if there is a probability of an AI evolving to 'think' that elimination of the human race is a logical path then isn't it equally likely that there will be another AI evolving logical paths to preserve the human race?

I have two competing responses:

1) Check out the AI of Iain M Bank's Culture series [0]. In it is a benevolent society of AI machines (the Minds) that generally want to make the universe a better place. Shenanigans ensue (really awesome shenanigans).

2) In response to the competing AI directives, I'll reference another Less Wrong bit o' media, this time a short story called Friendship is Optimal [1]. Wherein we see what a Paperclip Maximizer [2] can do when it works for Hasboro. (It is both as bad, awesome, and interesting as you might expect it to be.)

Personally, I think the general idea is that once one strong AI comes about, there will also be a stupefying amount of spare idle CPU time available that will suddenly be subsumed by the AI and jumpstart the Singularity. Once that hockey stick takes off, there will be very little time for anything else to get in on being a dominant AI. It's... a bit silly written like that, but I get the impression it's assumed AI will be just like us: both competitive and jealous of resources, paranoid that it will be supplanted by others and will work to suppress fledgling AI.

I have no idea why this is prevailing, aside from it's like us. Friendship is Optimal makes a strong point that the AI isn't benevolent, merely doing it's job.

> The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else. — Eliezer Yudkowsky

[0] http://en.wikipedia.org/wiki/The_Culture

[1] http://lesswrong.com/lw/efi/friendship_is_optimal_a_my_littl...

[2] http://wiki.lesswrong.com/wiki/Paperclip_maximizer

EDIT: I feel it may be appropriate for me to share my opinion: AI will likely be insanely helpful and not at all dangerous. But there will be AI that run amok and foul things up - life threatening things, even. But we already do that with all manner of non-AI equipment and software, so I'm not terribly worried (well, no more so than I usually am).

Any intelligent animal is fighting for its survival when feeling threatened. There's no reason to assume that a self-aware AI will be OK with us simply pulling the plug on it.

I'd like to think we could find some middle ground ebtween helpless surrender and imposing the death penalty on a sentient individual, both in moral terms and in terms of having some failsafe mechanisms, so that supplying electricity didn't allow for a takeover of the power grid or other doomish scenario.

I think bostrom/Yudlowski's arguments are a bit flawed on thsi topic.

The AGI would improve its intelligence, not because it values more intelligence in its own right, but because more intelligence would help it achieve its goal of accumulating paperclips.

Why is the worthiness of this goal not subject to intelligent analysis, though? The whole scenario rests on the idea of an entity so intelligent as to wipe out all humanity, but simultaneously so limited as to be satisfied with maximizing paperclips (or any other limited goal for which this is a proxy).

An AGI is simply an optimization process—a goal-seeker, a utility-function-maximizer.

Then I submit that it's not an artificial general intelligence because it apparently lacks the ability to evaluate or set its own goals. I'm reminded of the 6th sally from the Cyberiad in which an inquisitive space pirate is undone by his excessive appetite for facts.

>it apparently lacks the ability to evaluate or set its own goals.

The AI would have to evaluate the goal by some standard, so 'maximize paperclips' is a proxy for whatever goals get a high evaluation from the standard. Getting the standard right presents essentially the same problem as setting the goal.

Putting in 'a need to be intellectually satisfied by the complexity of your end product' is complicated and still wouldn't save humanity.

As shortly as I can muster: we live in a world with finite resources. Any consumer of these resources for its own purposes, whatever they may be, is in direct competition with us (all human civilization). Any entity better equipped to gather and make use of these resources will leave us resourceless. The space of human goals is a very tiny and constrained sliver of motivation-space, so by default AI goals fall outside of it.

To quip: The AI does not love you, nor does it hate you, it simply does not care. You are made of atoms that it can use for something else.

It feels like both you and the previous responder are predicating that there can only be one AI. If there is only one AI then yes, they don't love/hate or care about people and will likely have the resources to use them for their own means.

But, and I am looking for math, science or something more than Sci-fi, that can can show us that there will likely only be one AI, ever. If not wouldn't the AI's also try to manipulate and exploit each other for individual gains?

Perhaps I just don't understand AI well enough but I have yet to see any reasonable evidence that points to only one AI entity evolving on earth. If there are more than one AI then is in unreasonable to think that people may still be able to think thoughts or come up with (emotional and irrational ideas) that the AI could not come up with that give it a competitive advantage over the other AI's? Thereby leading to a more symbiotic relationship between humanity and the AI's.

AI is both singular and plural. The important difference is whether there are zero or non-zero, irrespective of their relative competitions.

Human intelligence took millions of years to evolve. Sorry, but AI(s?) will develop faster.

> Human intelligence took millions of years to evolve. Sorry, but AI(s?) will develop faster.

Just because it happens faster doesn't mean that the underlying laws that guide evolution in the universe are less applicable.

>Are there not just as many paths towards protective, helpful <or insert one of many adjectives here> AI futures?

No. "Bad" is just the state of the universe by default. "Good" is an extremely small island in a sea of "bad".

> "Just to be absolutely clear, we don't know how most kinds of thoughts are represented in the brain."

I've spoken about this recently. It's important that we keep in our minds a particular distinction about how we teach and approach things in Machine Learning and Neural Networks: that is, we are creating computing architectures that emulate a model for how we think the brain might operate, and we've achieved some non-trivial successes in doing that. We can't forget that it is only a model that allows us to develop some intuitions about how the algorithm works, but decidedly not how the brain works.

> "Today, the absence of knowledge is a fragile and temporary condition, like the darkness in a closet whose door happens to be shut." - Eliezer_Yudkowsky @ LessWrong

I gotta say, I really like the idea that a lack of knowledge is a fragile thing. Too often, we imagine ignorance as a large, immovable barrier. And the condition is usually hard to dissuade, but we certainly can progress onward in spite of it with hardly the effort of simply ignoring it. Like cracking a door open slightly to look in.

Reply on news.ycombinator.com