Leakage and the reproducibility crisis in ML-based science
reproducible.cs.princeton.edu
reproducible.cs.princeton.edu
pure science that is publicly funded in the public interest would publish all raw data along with re-runnable processing pipelines that will literally reproduce the figures of interest.
but, the funding is often provided by governments with the aim of producing commercializable new technology that can make life better for society.
the problem is that if you do the science in the open, then it can be literally picked off by large incumbents before smaller inventors have a chance to try and spin up commercialization of their life's work.
so we have this system today where science is semi-closed in order to protect the inventors, but sometimes to the detriment of the science itself.
1) How do you handle research in domains where the data is about people, so that releasing it harms their privacy? Healthcare, web activity, finances. Sure you can try to anonymize it, anonymization is imperfect, and even fully anonymized data can be joined to other data sources to de-identify people; k-anonymity only works in a closed ecosystem. If we live in a world where search engine companies don't publish their research because of this constraint, that seems worse than the current system.
2) How does one define "re-runnable processing"? Software rots, dependencies disappear, operating systems become incompatible with software, permission models change. Does every researcher now need a docker expert to publish? Who verifies that something is re-runnable, and how are they paid for it?
while it's not great for science at large, i don't blame them either.
For example, ImageNet for machine learning is a very expensive and difficult data set to produce that has resulted in revolutionary advances in machine learning. And people build models on it, cite their results as evidence their models are good, and cite the paper.
that's an interesting problem that i have not thought about.
i think maybe that this is not a technical problem, but more an ethical one. under the open data approach, if you want to study humans you probably would need to get express informed consent that indicates that their data will be public and that it could be linked back to them.
> 2) How does one define "re-runnable processing"? Software rots, dependencies disappear, operating systems become incompatible with software, permission models change. Does every researcher now need a docker expert to publish? Who verifies that something is re-runnable, and how are they paid for it?
one defines it by building a specialized system for the purpose of reproducible research computing. i would envision this as a sort of distributed abstract virtual machine and source code packaging standard where the entire environment that was used to process the data is packaged and shipped with the paper. the success of this system would depend on the designers getting it right such that researchers _wouldn't_ have to worry about weird systems level kludges like docker. as it would behave as a hermetically sealed virtual machine (or cluster of virtual machines), there would be no concerns about bitrot unless one needed to make changes or build a new image based on an existing one.
the good news is that most data processing and simulation code is pretty well suited to this sort of paradigm. often it just does cpu/gpu computations and file i/o. internet connectivity or outside dependencies are pretty much out of scope.
i don't think it's hard... there just hasn't been the will or financial backing to build this out right and therefore it does not exist.
As someone who wants science to advance, I want highly trusted researchers to be able to do studies that involve my private, personal data, that I would not consent to being public and linked back to me.
It is highly important to me that we allow these studies to not use open data.
A great example of this is the US college scorecard, which uses very private tax returns to measure how much college degrees and majors contribute to income (not the only value of college education, but certainly an important one):
https://collegescorecard.ed.gov/
Only high degrees of trust allowed this data to be published on extremely private information, and I think that makes for a better world. I am pro-open data, but research on non-open data should absolutely exist.
For instance, should any research about mental health for transgender people be abolished? Because anything on that subject is not going to be open, or at the least those who would be open to their data being public are a probably non-representative subset.
10~20 years ago I could see it. Nowadays it’s a tough ask that would severly limit the number of people participating. This could also steer away most minority groups, which would make the research not only limited, but also misleading (we’d still draw conclusions from them, and decide policies accordingly, even as they come from grossly biased participant pools)
Aside from just the public aspect of having ones data in the open, there is also second/third order discoveries that would happen from there (e.g knowing someone’s cooking habits could be enough to deduce overall health status, potentially chronic illness, ethnicity/religion, relationship status etc.)
This is always a problem even with some of the most open scientific code.
Requiring that the code be published, and perhaps a peer-reviewer to run it just once with a bit of support to ensure that the submitters aren't completely bullshitting, before the paper gets approved to be published, might be a good start.
I think in large parts it's a tooling and awareness problem - like, historically awareness and use of source control has been very low in the space, but for data, even industry tools tend to be less sophisticated than the ones for code.
And there are those who say progress proceeds optimally in this arrangement! Hardly: only "innovation", which is distinct from progress, is optimized. Progress is the act of cleaning up the mess made by ineffective innovations (ie "minimum publishable units"), distilling them into a basis for the next paradigm shift.
Progress, everyone's ostensible goal, is hampered by siloes and IP hoarding, as it multiplies the efforts made by many who should otherwise be working in tandem, by hiding them each in their own little silo so they all learn the same lessons completely separately.
That's how you get "limited protection" of the arts and sciences, and patents that were democratic ie I can file on my own for a provisional patent, it's really hard in cases like the Wright Brothers--and some others, and there's crackpots--but at least you didn't need to lobby to get one like in England, where it was an act of Parliament to get an inventor his patent, like James Watt. [Don't have visibility on how the two systems evolved when they broke apart.]
But rather than stand-alone, it should be incorporated into publications.
In my experience, only a minority of applied machine learning papers provide even a minority of the info requested by the info sheet.
Meaning, you really have no proper idea how cross validation was done, what preprocessing was done etc. - in actually published papers
Oh well, I guess this is only meant for insiders who understand the in-group jargon. The rest of us need not be interested at all.
Edit: for context, here is the paragraph titled "Leakage" from the draft paper:
Leakage. Data leakage has long been recognized as a lead- ing cause of errors in ML applications (Nisbet et al., 2009). In formative work on leakage, Kaufman et al. (2012) provide an overview of different types of errors and give several rec- ommendations for mitigating these errors. Since this paper was published, the ML community has investigated leak- age in several engineering applications and modeling com- petitions (Fraser, 2016; Ghani et al., 2020; Becker, 2018; Brownlee, 2016; Collins-Thompson). However, leakage oc- curring in ML-based science has not been comprehensively investigated. As a result, mitigations for data leakage in scientific applications of ML remain understudied
https://arxiv.org/pdf/2207.07048.pdf
Yes, alright. But what the flying fuck is "leakage"? Am I supposed to go read the "formative work on leakage" cited? What if that work also leaves it undefined and points to an earlier source? What the hell is this paper about? Hhow hard it is to explain what your main subject is, so I can read your paper knowing what you're talking about?
How frustrating.
There is a whole one and a half page section (2.4. Towards a solution: A taxonomy of data leakage) that describes every kind of leakage the authors considered for this work.
This is what will happen to most people who read the draft paper. They will start with the abstract, feel confused, get annoyed and leave before they get to the "texonomy".
In fact I did glance at the "taxonomy" as I was scanning through the paper dejectedly in the last few seconds I bothered with it and it still didn't seem to explain what the different kinds of data leakage are different kinds of. That's basically when I stopped reading.
And this is why you don't write papers like that, because it makes them less likely to be read. And because, as an author in a field where people put tens of thousands of preprints on the internet, you want to make absolutely sure that your paper is as likely to be read as possible, by as many people as possible.
Basically, if you want people to read your papers, the absolute worst thing you can do is to expect your readers to "keep reading to see what we mean". Most people will give up and reviewers will be so annoyed you're not respecting their time that they'll skim your paper looking for reasons to skewer it.
The next worst thing is to expect your readers to try and guess what you mean. I could very easily make an educated guess at what "data leakage" means, but then I would have to read the paper while never being sure I know what exactly I'm reading and whether I have it subtly wrong. Again, you don't want to cause that kind of stress to your readers. You want everyone who reads your paper to feel happy and calm and admire your bright, squeaky clean ideas.
So make things as clear as possible, as hassle-free as possible, as friction-free as possible. Otherwise, you're kicking yourself in the butt.
That's my free advice.
To not provide accurate ways of reproducing something claimed in a paper means that the paper is invalid.
It's a very well-defined problem and the datasets it uses are very well-characterized (the protein crystallography database, maybe some NMR structures as well), so perhaps that helps.
"we advocate for a standard where bugs and other errors in data analysis that change or challenge a paper's findings constitute irreproducibility."
Please no. One of the biggest problems I face when talking to people about bad science in any context is the belief that "peer reviewed and reproducible" is a synonym for correct. The phrase the authors are looking for here is not irreproducible, but rather something like: flawed, incorrect, biased, pseudo-scientific, or even intellectually fraudulent. The danger of trying to redefine the word reproducible to mean more than "do people get the same results" is threefold:
1. Researchers will reject claims their work is not reproducible by saying "actually they didn't do the same things we did so of course they didn't get the same results" and that will be a convincing rebuttal to outsiders who don't dig into the details.
2. It would further undermine trust in academic research. Way too frequently, I read a paper that makes an interesting claim, only to discover that their paper or maybe entire field has redefined common words in ways that make the claims misleading.
3. It doubles down on the unhelpful and probably counter-productive "reproducibility crisis" framing.
Why unhelpful, well, there isn't really a reproducibility crisis. What we have here is actually an intellectual fraud crisis. After reading a ton of papers from outside CS in past few years it became impossible to avoid the uncomfortable conclusion that in many fields the majority of observable errors are not really errors at all, but are actually deliberate. Or at least, they are deliberately fooling themselves which amounts to the same thing.
To highlight just a few examples of mistakes where you think, how can nobody have noticed this:
• (from the linked paper) "a recent study included the use of anti-hypertensive drugs as a feature for predicting hypertension. Such a feature could lead to leakage because the model would not have access to this information when predicting the health outcome for a new patient. Further, if the fact that a patient uses anti-hypertensive drugs is already known at prediction time, the prediction of hypertension becomes a trivial task"
• A widely used ML model from social science that claimed to predict if a Twitter account is a bot was put online and found to have an FP rate of 50%+. When this was pointed out by third parties, the response was to claim the testers were "academic trolls". Nothing was ever retracted and the model continued to be used in new papers across the field.
• A COVID modelling paper blithely computed that the average Brit lives with 7 people. This was obviously wrong both in absolute values and just being nonsensical design to begin with (that should be an input taken from census data not an output), and the peer reviewer even noticed this but approved the paper anyway.
• The Ferguson Report 9 model that directly led to lockdowns in the UK and other countries, was full of computational bugs like typos in their custom PRNG, out of bounds memory reads, bugs in a custom Fisher-Yates shuffle, thread safety errors and more. Nobody in the academic world appeared to care about this.
The paper authors suggest making researchers fill out more paperwork to get published. I find it impossible to believe that this will work. Mandatory signed data sharing statements didn't work: there was a study posted on HN in the past few months in which someone tested this to see if the data was genuinely made available on request and something like >90% of scientists refused (in epidemiology I think). Similar results were found in other fields. In this light the mass adoption of ever more opaque and buggy statistical/computational techniques is not merely an accidental drift, correctable with a minor bit of bureaucratic oversight. These techniques seem to be popular exactly because they grant so many angles of freedom to get away with scientific murder.
So of course there are cool things like gpt, but it's not like it's scientific progress. It doesn't really to understand how brains work, and how to understand what general intelligence really is.
However, ML is useful to generalist science as long as you are be aware of its shortcomings and not just trying to replace something with ML without thinking about it.
To give you an example I worked on (to be published): I worked with some physicists that use an incredibly slow and expensive iterative solver to get information on particules. We introduced a machine learning algorithm that predicts the end result. It does not replace the solver (you could not trust its results, contrary to a physics based numerical algorithm) but, using its guess as a starting point for the iterative solver, you can make the overall solving process orders of magnitude faster.
And I guess the outcome variable in the train set for the ML model was produced by the solver?
It is also valid to make scientific progress just inside of a field and not in the grand scheme of things.