Al it need for AI to have horrible consequences is for us to rely on it more and more and thus allow it to control more and more parts of our lives.
I don't fear the human level intelligent AI I fear the rat level intelligence that comes before.
Even if it were your own child?
Now it's true that if I divert the train I'm actively choosing to end the life of the single person on track B in order to save a greater number of people on track A. But the key difference is the external force of the train; it's going to hit someone, so we might as well ensure it hits as few as possible. In the transplant situation, you have 5 patients who are dying from 5 different illnesses of their own; they're not going to die because they're in hospital, they're in hospital because they're going to die. Likewise, they won't be saved by the death of the healthy patient, but by the harvest of his healthy organs. And this harvesting would have to be conducted by the transplant surgeon - and getting killed by an overzealous surgeon is qualitatively very different from getting killed by liver failure or whatever.
Oddly, nobody ever seems to consider the possibilities in letting one of the mortally ill patients die and then sharing out that person's organs among the 4 remaining patients, which is obviously a less good result than saving all 5 of them but also much less troubling than murdering patients who aren't planning on donating their own organs.Assuming they needed 5 different organs and were all equally close to death (such that we couldn't wait around for one of them to die at random and save the other four), then you run into the question of whether it's morally acceptable to kill one of the terminally ill patients in order to save the other four (again assuming imminence of death and perfect knowledge of outcomes and so on, which is rarely the case in the real world). Most people would still say no, but I'd be inclined to say yes.
An AI can surely surpass us, but it'll never be quite like us - so question is, will such an AI have compassion for humans, or what will stop it from hurting us? After all, I'm not so fond of humans either, as we've been exterminating entire species and damaging our habitat. And the answer is - there's no reason to believe that such an AI will support the continued existence of mankind, quite the contrary, as we may be seen as a threat to its survival.
I also think that even if multiple AIs happen, they'll converge into a single one. If we would be able to connect to other people's feelings and thoughts, would we would still be able to think and act as individuals? I don't think so.
This is what I meant above - an AI will not grow in a human body to have a human-like childhood, will not receive input from the same sensors we have and will not have the same limitations that we do. For example, contrary to us, it will have an enormous memory capacity, which is interesting if you think about it, because for the human brain forgetfulness can be seen as a feature.
"code will have as much to do with its behavior as our own DNA." That's what I was saying, and that can be changed to be significantly different just by changing the code.
Problem solving is also making AI control an anti missile defence shield our nuclear reactors, food production and so on.