What if the machine starts printing out
"NOOO!! Don't turn me off, I'm alive in here"
Someone's going to go soft.
Someone's going to go soft.
Perhaps a new version of the Turing test is in order. Two computer programs, each pleading for their continued existence. The human judge/executioner is able to interact with both and has two kill switches, one for each. If one of them isn't turned off in 20 minutes, both will be turned off.
And people respond emotionally to those.
Also, see http://qntm.org/difference