Sloppy Use of Machine Learning Is Causing a ‘Reproducibility Crisis’ in Science

"Reproducibility crisis" is the meme label that gets attention these days, but I feel like what's happening here is different enough to deserve its own name.

When a study is not reproducible, it means the authors didn't provide sufficient accurate methodological detail for reviewers to reproduce the study results by reprocessing the raw data according to the authors' declared method.

In the cases described here, it sounds like the reviewers were able to reproduce the results - it's just that, once they dug into the details, they found the method used contained subtle but catastrophic flaws.

The crisis here is one of methodological validity. Use of machine learning in research greatly increases the risk of data leak, among other things. The scientific community, both authors and reviewers, will need to become more aware of these risks and proactively point out when results may be subject to such issues.