https://www.youtube.com/watch?v=_lp9-cN_Oog&t=4m23s
> These men were able to give the counsel they gave because they were operating at an enormous psychological distance from the people who would be maimed and killed by the weapons systems that would result from the ideas they communicated to their sponsors. The lesson, therefore, is that the scientist and technologist must, by acts of will and of the imagination, actively strive to reduce such psychological distances, to counter the forces that tend to remove him from the consequences of his actions.
-- Joseph Weizenbaum
But what happens instead is often enough that this "warm, fuzzy" feeling we get from that (awesome) story, is also evoked for the "awesome technology that can make a real impact to save human lives" or "make wars less bloody" etc.
Bottom line is, it cannot think. Even humans too often don't really think about what they are doing, but our machines cannot, even in their brightest moments. Not on the level we're talking about here, anyway. If they could, we would make damn sure it could not "refuse orders".
That is not a bug, but the feature. And one doesn't build a cannon to then not shoot a cannon ball.
"Dark Star" comes to mind:
She turned and faced him, and lightly held both his arms. He was a picture of bewildered misery.
"Tell me," she said.
He said nothing for a moment, while his gaze darted from one of her eyes to the other.
"We ..." he said, "we have to be alone ... I think." He screwed up his face and then dropped his head forward, shaking it like someone trying to shake a coin out of a money box. He looked up again. "We have this bomb now, you see," he said, "it's just a little one."
"I know," she said.
He goggled at her as if she'd said something very strange about beetroots.
"Honestly," he said, "it's very, very little."
"I know," she said again.
"But they say," his voice trailed on, "they say it can destroy everything that exists. And we have to do that, you see, I think. Will that make us alone? I don't know. It seems to be our function, though," he said, and dropped his head again.
"Whatever that means," said a hollow voice from the crowd.
Trillian slowly put her arms around the poor bewildered young Krikkiter and patted his trembling head on her shoulder.
"It's all right," she said quietly, but clearly enough for all the shadowy crowd to hear, "you don't have to do it."
-- Douglas Adams