Peter Thiel writes about how fatal machine learning for fraud detection in his book, "Zero to One".
At Paypal, Max Levchin assembled an elite team of mathematicians to study the fraudulent transfers in detail. Then we took what we learned and wrote software to automatically identify and cancel bogus transactions in real time. But it quickly became clear that this approach wouldn't would either. After an hour or two, the thieves would catch on and change their tactics. We were dealing with an adaptive enemy and our software couldn't adapt in response.
The fraudsters adaptive evasions fooled our automatic detection algorithms, but we found that they didn't fool our human analysts as easily. So max and his engineers rewrote the software to take a hybrid approach: the computer would flag the most suspicious transactions on a well designed user interface, and human operators would make the final judgment as to their legitimacy.