Comment by xerophtye | Hacker News Reader

Comment by xerophtye | Hacker News Reader

Parent Full thread

While i really like your point about "lack of motivation", here's how i think an "evil AI" scenario might play out:

You design a super system that has access to a shit ton of data and it can interface with pretty much every device on the planet. Let's suppose you gave said AI the task of maximizing your portfolio's profit. Based on the boat loads of data it has, it figures out that stocks of companies plummet when a key figure in the company dies. It identifies the competitors of the companies in your portfolio. It identifies the key players in those companies. It concludes that if these people were dead, their companies would crash and your profit would increase. It uses knowledge of how humans die, and use that combined with devices it can interface with, and eliminates the competition.

Yes, yes, seems too far fetched (i feel weird talking about this too). But see, it's possible. of course no one thought of teaching a financial software that killing people is bad. No one foresaw that it would figure out that this was a way to maximize profit. No one saw that the interfaces we provided it to study consumer behaviour and correlate with stocks, can be used to actually influence said systems to take human life.

Is this a computer turning evil? No. Is it connected to some sort of motivation for self-preservation or "eradicate man kind" or some human emotion of jealousy? No. It was just a logical result from given data and task, on which the computer was able to act.

You're planning to have an AI that's smart enough to figure out how to pull off a murder spree, but stupid enough to think it's a good idea for your portfolio?

And it's going to hack into and remote-control peoples' Teslas (or whatever) to pull it off because you can just say "it's an AI, it can just figure out how to do that sort of thing!" without regards to the computational expense of breaking that sort of security?

You're right, it is a little far-fetched.

An AI would not be 'smart enough' to figure out what is a good idea or not. That requires a human context, that's the whole point. An AI is not good or evil, its ultimately a loose cannon with no understood (or even understandable) motivations at all.

Sure, but if an AI isn't smart enough to figure out "trying to murder corporate leaders is a bad plan which could lead to seizure of the portfolio I'm trying to manage, among other things" then how do you expect it to be smart enough to engage in the sort of open-ended planning necessary to pull off the described murder spree? :)

As someone pointed out, perhaps the example wasn't thought out well enough. So why go into the details of that, and skip the underlying point? That maybe the system WILL come up with a bad plan and execute it? Or are you saying:

"if the system is smart enough to plan murder, he is also smart enough to know murder is bad" ??

Doesn't that depend on what data the system has seen? Perhaps he doesn't he doesn't have cases in its knowledge base that demonstrate eh side effects of murdering business heads. Perhaps he can only see what happens when they normally die? What if it simply correlates the wrong variables (and forgets correlation isn't causation") ??

I apologize for coming back to the same example again. but it would be really nice if you overlook the flaws of the example and discuss the underlying point

the example might not be well thought, but it conveys the basic idea that the motivation might be a side effect of some rule in the system that plays the role of the evolutionary fitness function.

We think in terms of physical survival, but we can use an arbitrary fitness function to train a genetic algorithm.

Hey, the first thing that crossed my mind when i opened this post was "why the hell are we even discussing this? Don't these people realize what sort of computational improbability it is at this point?"

So yes, this very discussion is EXTREMELY far fetched. So any statements I make here, come with "assuming we actually HAVE a system capable of performing such tasks".

Hence this comment was in response to my parent's comment that "Even if we have such powerful machines, what would be their motivation for doing something like this?" hence i gave my opinion that perhaps it is just trying to perform a task it was told to do, and based on available data, its strategy includes crime. So perhaps you may argue that this would be an inefficient strategy. Perhaps you are right. But none of us can be really sure that the system would see it that way too. After all, how many of us have never seen a program do something it wasn't intended to do?

I am just saying that maybe the proponents of this discussion have some point. That perhaps giving too much power to systems is bad. (Assuming we CAN give them this much power)

Reply on news.ycombinator.com