Hacker News Reader

Comment by KVFinn | Hacker News Reader

>Wouldn't a drastically more advanced intelligence just dwarf and dissolve any lesser intelligence that tried to meld with it? It would be complete ego death.

A lot of the singularity people think that superhuman AI in inevitable whether they work on it or not, but if the first AI isn't a friendly one (as opposed to a paperclip-optimizer that happily turns the planet in paperclips) then you don't get a second chance.

View on HN

Also called "a self-fulfilling prophecy"; you make something happen by the simple statement "it will happen".

It's also a smart move for your career: You can create your own job and attract lots of funding for your research, or sell your business for a lot.

It sounds like a piece of fearmongering nastiness until you think of it in terms of mathematics rather than psychology. A so-called "artificial intelligence" is just an extremely sophisticated active, online learning agent designed to maximize some utility function or (equivalently) minimize some loss function. There's no term in a loss function for "kill all humans", but neither is there one for "do what humans want", or better yet, "do what humans would want if they weren't such complete morons half the time".

Or as Eliezer nicely phrased it once in a paper[0],

> "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

[0] - http://yudkowsky.net/singularity/ai-risk

This objection implies there is something better than turning the planet into paperclips.

Reply on news.ycombinator.com