Ethics of AI
vimarsh.info
vimarsh.info
This sounds similar to Asimov's Three Laws of Robotics[1]:
First Law: A robot may not injure a human being or, through inaction, allow a human being to come to harm.
Second Law: A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
Third Law: A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
Asimov wrote a lot of stories exploring the consequences of these laws, and thinking up the many ways in which they could be circumvented.
I've also heard of a parody of Asimov (I forget by whom), featuring a robot with a faulty human detector.
The point is that such safeguards are likely not going to be nearly enough to keep humans safe.
Some people don't think strong AI should be developed at all, but I doubt humanity is going to be effective in stopping it. It's just too tempting because of the potential to give its creators incredible power, also because of natural human curiosity, and possibly for religious reasons (as these entities may be seen by some either as gods themselves or as ways to contact gods).
As technology advances and the possibility of developing strong AI comes closer and closer, and within the grasp of more and more people, it's likely that someone's going to do it.. if it's possible at all.
After that, it's hard to predict what's going to happen, as the singulitarians made clear.
This is the only reason AI hasn't exploded yet (we know we're playing with fire with AI). It's called our 'final invention' for a reason.