Dos and Don'ts of Machine Learning in Computer Security [pdf]

I guess one hypothesis is that security companies are just doing a really bad job of it in the rush to get products out of the door with the magic "machine learning" tickbox on it.

That would not surprise me...

I work in ML research and you're correct that none of this is new or surprising. All pitfalls are common knowledge in the field and not specific to security. But the security industry is still lagging behind here and the paper does give some nice examples for how these pitfalls can easily be stumbled into if you are not careful during the creation of datasets, training and evaluation. Curiously though, their Android App risk example lists the origin of the app under spurious correlations, whereas in practice any careful human would also immediately rank an app from a random chinese store as a greater risk than one from the google play store. This is part of the problem: There is a large gap between academically worthwile discussions and the stuff that works in the real world.