The danger comes from our inability to accurately define our own desires. There is no doWhatIMean() function. A programmed goal can subtly differ from the one we want so that it passes the obvious test cases but breaks catastrophically in an unusual real-world situation. There's an anecdote[1] about a neural network trained to detect camouflaged tanks (I've also heard of it detecting Soviet versus NATO tanks); it ends up learning to detect objects with shadows, because the tanks were photographed on a sunny day and the non-tanks on an overcast day. Now imagine if the program had been prematurely instructed to attack the camouflaged tanks. You can call this "broken and useless software," but by the time we realize how it's broken it could be too late.
[1]: http://www.j-paine.org/dobbs/neural_net_urban_legends.html