I remember reading a blog post (if someone has the link, that would be much appreciated) of an Nvidia developer who was training a model to play Pac-Man a few months ago, and they simply told the AI to avoid death however possible. They let it run through a sizable amount of iterations with expectedly poor performance, but the model eventually "learned" that it could pause the game to avoid a failure condition. Since it was one of the most successful iterations around, the entire model eventually had to be re-trained without pausing enabled to force it to perform better.

I think the ultimate moral of this story is to be careful what you wish for. We should be excited at the opportunities that AI presents, but baby steps are important right now. As this article points out, even basic image classification can be fooled with nothing more than a pen and paper. It seems a little dangerous to be incorporating this into anything mission-critical, and I suspect we'll soon see a rise in "hacks" that involve exploiting bias in machine learning.