140 karma · joined December 16, 2016
If you owe the bank $1 billion dollars, you own the bank.
https://wiki.lesswrong.com/wiki/Paperclip_maximizer
One of the reasons AGI will be so dangerous is the near certainty it will have utterly alien values and intentions.
It seems exceedingly unlikely that we will be able to program in what consitutes "human values". We can't even begin to remotely agree and explain them fully to fellow humans, let alone what is basicly an alien intelligence.
It is slightly ironic that people bring up Assimov's laws of robotics in this context, because pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI.
If something is very much smarter than humans, we cannot possibly hope to constrain it via some kind of faustian bargin agreement. Because in the end, the devil is always smarter than you are.
If an AI kills you from either malice or indifference, or just that you are composed of plenty of paperclip-capable raw materials, you are dead all the same.