Is the Reproducibility Crisis Reproducible?
argmin.net
argmin.net
A z score is not at all morally equivalent to a p-value. It's just a standardized measure. Converting measures to z-scores aids in interpretation. They also can aid estimation in some cases: using non-standard parameterization in Bayesian analysis is often crucial to get MCMC to accurately sample from the posterior distribution.
Sure, you can take a z score and look at the area under the curve and come up with a p value. But you don't have to. In the referenced paper, they use z scores to be able to standardize the measures in the papers they draw from, so they're comparable.
The author's other critiques of the paper seem reasonable. It's a problem with all meta analyses: the amount of work it takes to correctly interpret publishes papers and then take those results and aggregate them is herculean. To do it to over 20,000 is inevitably going to lead to some mistakes. That said, those mistakes may not be fatal to the analysis.
Moreover, saying "no one knows what happened to those 11,285 studies" without checking in with the authors is completely unfair. The first author responded with the code showing exactly how they achieved that figure. Nothing mysterious.
Andrew Gelman responded in the comments to him, as did the first author. I find their responses convincing.
But that is the whole point! The methodology of how they dropped the 11,285 studies was not even told in the original paper, and even in the comments the author doesn't explain "why", just "how". Hence, I think it's completely fair to call it "irreproducible".
The point of doing "reproducible science" is not that I write a paper, you email me asking how did I come up with my number, and I email you back an explanation. No! The important details should be in the paper already. You may do some magic on your dataset, and that's fair _as long as you detail what magic you did and why_, and given you can defend that practice in front of your peers. Otherwise what is the point of preaching "reproducible science" at all?
This is how it works, people email each other all the time. Why shouldn’t you? You can’t imagine every bit of information that someone would want, and papers have page length limits so you make choices about what to cut.
If Bob falsifies data and someone e-mails him, asking him to send them rope to hang him with, he can simply delete the e-mail.
Or claim he forgot the details of the analysis. Or claim it was handled by a grad student who left. Or claim the info was lost when a hard drive broke. Or claim the data was the intellectual property of College A and he can't access it now he's at College B. Or claim privacy or copyright rules cover the key data. Or that they don't have a license to the software that can open the data files any more. Or any of a dozen other things.
Typically bad research assumptions or implementations are rooted out during peer review, but it’s an imperfect process.
I do think there needs to be a dedicated non-profit and neutral organization solely responsible for reproducing scientific results for all fields, and assigning a reproducibility score to research finding. This could become an entire field by itself, and would have its own complications, but the reproducibility crises does exist and needs a solution.
Otherwise, years later, if the author dies, the paper would be basically worthless for reproduction.
(We're still talking about reproducibility crisis, right?)
Maybe it sounds like I'm parsing too much, but I nevertheless still think saying "no one knows what happened" is unfair. They know, they shared, and justified what they did when called on it with literally almost no delay. I agree they shouldn't need to be called on it.
Anyone who's advised students or asked even presenting researchers such questions know that often people will literally not know what happened to all their data.
I am sorry that’s been your experience, maybe it varies by field and quality of research? Most people I’ve questioned have provided reasonable answers to their findings. I don’t understand why anything needs to be assumed in bad faith or shoddily done. It’s all a bit Dunning-Kruger to me where everyone assumes that everyone else is doing shoddy or bad work.
To be fair: Everyone I've worked closely with in research has gone above and beyond not to cut corners and produce high quality data and research.
What I have in mind here is a situation where people are actually quite careful but can still end up in a place where they don't know what happened because they don't have good systems for creating datasets and storing code.
For example, graduate students are not always taught to work in a reproducible way. It's definitely gotten better from what I can see, but it was normal for people to get source data and work that data into its final form in a lot of different steps, but not always reproducible steps. E.g., data comes in from secondary source or other provider. It gets cleaned. That file gets saved as something like "clean data 011234.csv".
More work is done, it gets saved again.
Time passes, things are revisited, and a handful of files exist that likely with some care could lead from point A to point B. But the exact process, to say nothing of the dozens, sometimes hundreds, of small decisions data preparation decisions get lost to memory.
Code doesn't go in version control. People get new computers. USBs get lost. Universities migrate to new data systems and so on.
All the while, these students and researchers were very careful while doing the work. They were just never trained to use good version control and pipeline processes. They basically do what they did with papers they write. Save and backup while working through the paper and move on when it's done.
This is made worse when data is proprietary or not legally shareable.
So people aren't necessarily being shoddy or doing bad work, they're just not using good systems.
Agreed. I think there isn’t an incentive to do this because reproducibility takes a back seat to so many other concerns. Unless PIs are told that their publication chances depend on reproducibility, this isn’t going to change.
Fantastic article on the reproducability of Dunning-Kruger effect: https://replicationindex.com/2020/09/13/the-dunning-kruger-e...
Also see: "The Dunning-Kruger Effect is Autocorrelation" https://economicsfromthetopdown.com/2022/04/08/the-dunning-k...
Lovely quote:
“These responses to our work have also furnished us moments of delicious irony, in that each critique makes the basic claim that our account of the data displays an incompetence that we somehow were ignorant of.” (Dunning, 2011, p. 247).
The below seems like a "why" to me.
> The criteria for selecting the data are an atttempt to get the primary efficacy outcome, and to ensure that each trial occurs only once in our dataset. The selection for |z|<20 is because such large z-values are extremely unlikely for trials that aim to test if the effect is zero.
Even if it does qualify, It's widely accepted that argument from authority is perfectly valid and often necessary when performing inductive reasoning.
I might even say that if you are not expert enough to judge the matter yourself (or willing to expend enough effort) it should be your default presumption that the more qualified speakers are correct, even if you have legitimate questions about their potential motivations.
This whole affair is a case in point. Several authors ended up responding with very reasonable responses, which the author then acknowledged. Then he writes another post, which is more measured and insightful.
The original post would have in fact been a far better one if the author had first sought those responses and just addressed them all together at once. His points about metascience would not then be dragged down by back and forth in the comments that clearly got personal.
For a sample the equivalent is the t-statistic, which indeed IS very often used for p-values (with the ever popular t-test) and has a decently strict list of assumptions (which, like those of [O|W|G]LS are very frequently ignored).
"By 2005 or so, it will become clear that the Internet's impact on the economy has been no greater than the fax machine's." -- Paul Krugman ( Nobel Prize Winner in Economics )
If you have to pathetically appeal to authority right from the start, you probably have no argument worth considering.
https://www.argmin.net/p/is-the-reproducibility-crisis-repro...
https://www.argmin.net/p/is-the-reproducibility-crisis-repro...
https://www.argmin.net/p/is-the-reproducibility-crisis-repro...
Also, I don't think the arguments written in the post are "unserious", you might not like them but the author of the blog makes some interesting points against the original paper.
[0]: https://www.argmin.net/p/arbitrage-in-data-exchange-rates
We are unable to reproduce measurements of the Gravitational constant.
Reproducibility - distinct from the claimed uncertainty by the experiments - has not advanced at all since the 1940's. This is unlike any other physical constant where precision and reproducibility has consistently progressed along with our technology:
https://en.wikipedia.org/wiki/Gravitational_constant
> Published values of G derived from high-precision measurements since the 1950s have remained compatible with Heyl (1930), but within the relative uncertainty of about 0.1% (or 1,000 ppm) have varied rather broadly, and it is not entirely clear if the uncertainty has been reduced at all since the 1942 measurement. Some measurements published in the 1980s to 2000s were, in fact, mutually exclusive.[7][30] Establishing a standard value for G with a standard uncertainty better than 0.1% has therefore remained rather speculative.
Recht should probably correct himself here (maybe by writing another article in collaboration with Gelman etc.), and people should be okay with other people making mistakes, and accepting apologies or mea culpas. Information is complex, and there are very few geniuses capable of understanding or synthesizing everything in isolation. It’s best to assume good faith work and collaborate to promote collective understanding.
Edit: ah okay, so this is related to politics somehow, and expectedly the discussion is devoid of reason. Or sense, apparently—see one comment in the blog where “the consensus is that there is no consensus”, whatever the fuck that means.
Honestly, that's enough red flags to dismiss this blog post as "not convincing enough, more work needed".
It’s interesting that this is basically the same message he was attacking. He and Gelman and most other bright people agree, don’t trust most published research. Of course this is especially true for meta-research as the author seems to suggest.
You start by not allowing interests with a financial incentive to get a positive result from a clinical trial to run those trials - it's an obvious conflict of interest. However, the University of California has been a long-time leader in the exclusive licensing and privatization of discoveries made with taxpayer dollars, and so the UC also has an incentive to promote bogus clinical trials because it gets a percentage of sales from UC-licensed inventions, and so a professor at the UC who writes a screed defending these practices (which is what this article amounts to) will thereby ingratiate themselves with the administration - positive career move!
The real problems come down to deliberate systematic bias in experimental design, which people outside of academia might consider to be fraud - and it should be well-understood that statistical tests have a very hard time revealing such systematic biases.
Let's take a specific example - Gilead's remdesivir, and Gilead's fellowship programs and gifting to UC Berkeley professors. The corporatization of academics in the United States means that trust in the independence of academic researchers is no longer warranted, they're more in the cheerleading section now than in the rigorous science section. Thus (2021):
https://pubmed.ncbi.nlm.nih.gov/34252308/
> "Here, we critically evaluate the assumptions of the models underlying remdesivir's promising preclinical data and show that such assumptions overpredict efficacy and minimize toxicity of remdesivir in humans. Had the limitations of in vitro drug efficacy testing and species differences in drug metabolism been considered, the underwhelming clinical performance of remdesivir for both COVID-19 and Ebola would have been fully anticipated."
[1] As others have pointed out, there are rebuttals in the comments from some of the authors of the paper; make of them what you will.
The "reproducibility crisis" / "replication crisis" is about #2: lots of published work, it seems, reports results that other people trying to do the same experiments can't reproduce. This probably means that their results, or at least a lot of their results, are wrong, which is potentially a big deal.
The article's complaint about the paper by van Zwet, Gelman et al isn't that. It's that some details of the statistical analysis aren't reproducible in sense 1: the authors did some pruning of the data but apparently didn't explain exactly what criteria they used for the pruning.
You could argue that actually #1 is worse than #2, because maybe what the authors did is bad but you can't even tell. Or you could argue that it's not as bad as #2, because most of the time (or: in this specific case) what they do is OK. But I don't think it makes any sense to suggest that they're the same thing, that there's some great irony if a paper about #2 has a #1 problem. That's just making a pun on two completely different sorts of reproducibility.
(Note 1: The article makes another complaint too, which if valid might be a big deal, but it doesn't have anything much to do with reproducibility in either sense.)
(Note 2: One of the authors of the paper does explain, in comments on the OP, exactly how they did the pruning. I have not checked what the original paper says about that; it doesn't seem to be available freely online.)
(Note 3: Even if the Zwet/Gelman/... paper were absolute trash, I don't really see how that would make the reproducibility crisis irreproducible. This is one paper published in 2023; the reproducibility crisis has been going on for years and involves many many unsuccessful attempts at replication.)
This reminds me of the brilliant Portal exchange between two AIs:
In the 2011 video game Portal 2, artificial intelligence GLaDOS attempts to use the "this sentence is false" paradox to kill another artificial intelligence, Wheatley. However, lacking the intelligence to realize the statement is a paradox, he simply responds, "Um, true. I'll go with true. There, that was easy."
In reality they don't fix anything, and probably make things even worse. There are no commonly accepted or analyzed criteria for assessing Bayesian analyses. Heck, an analysis being "Bayesian" doesn't tell almost anything about it as it can mean a plethora of unrelated things.
The crises of science are sociological, not technical. My take of the core issue is the changing of science from an ethics and reputation based institution into a neoliberal bean counting paper industry.
Saying it was an "ethics ... based institution" is challenging for me - it implies right and wrong, which requires someone to decide what is right and what is wrong.
Saying it was "reputation based" is even worst - lots of things got delayed because "reputable scientists" were not ready to embrace a change of paradigm.
Sciences as a system was and is constantly evolving and had many issues before and it has many issues now. Generally it evolved in a direction to maximize knowledge acquisition, and had to deal with the social structures at that time (ex: don't get at stake for you theories). If it is in a bean counting phase might not be that bad in the grand scheme of things, but should not get stuck here either.
Ethics here don't mean anything metaphysical. They mean something like "professional ethics" or even "taking pride of your work". More generally some shared (and maybe internalized) view of what's good/acceptable/bad behavior in a community. E.g. in medicine this is/was sort of codified in the hippocratic oath.
In such a system reputation stems (at least ideally) from how well the actors (individual or institutional) behave in terms of these ethics. This of course has its own set of problems and may lead to (other) forms of corruption. But IMHO it's less bad than the neoliberal model where individual actors should focus on their individual career success "by any means necessary" and just trust that "science is self correcting".
Frequentist hypothesis testing falls somewhere in the middle, and as long as it's not over-interpreted is fine as it is, with the advantage being it's easy to do. Though the downside is that it's not easy to interpret.
The downside of Bayesian statistics, in my experience, is that it's really hard to teach it to less-than-very-bright people, even in the academy. I mean, those who don't even understand what "falsifying a null-hypothesis" means (and doesn't mean) will have a very hard time doing Bayesian analysis properly.
In a more advanced setting, though, Bayesian approaches provide a much better tool for comparing a set of alternative hypotheses. But it requires that users understand the math, and are not just following some script, and also that those involved are willing to provide their priors before evaluating the data.
Is this an actual stereotype that Bayesians are right wing? It doesn't seem at all political to me, but I guess I've never met someone who self identified as a Bayesian either.
[0] https://open.substack.com/pub/argmin/p/is-the-reproducibilit...
I'm more of a Bayesian in interpretation of statistics and epistemology, and I'm pretty far left. N=1 of course.
Neoliberalism refers to a social order and doesn't have much to do with probability and statistics.
This can be contrasted to the preceding systems that had more regulation of markets and planned resource production and allocation. Roughly Keynesian or social democratic economics, with their associated view of social organization.
There has been a concurrent shift towards bibliometric assesment of researchers and institutions, to a large part to have metrics for the competition.
In general this is largely an application of the (neoliberal) New Public Management [2] model applied to academia and science.
Before this shift funding was based mostly on budgets akin to how e.g. schools or (public) healthcare and police are funded. Academics were mostly just hired to a (permanent contract) job when there was an opening. Anecdotally, my supervising professor was tenured almost straight after he got his PhD in the 1970s. And he didn't really have to apply for competitive funding until 2000's or so. And this was more or less the norm back then.
In angloamerican countries the neoliberal turn started already in the 1970s. Although there was also a contemporary explosion in funding of science much due to the role of scientific and technological progress in the Cold War.
The degree inflation likely plays a role too, and in research this shows as more PhD students who have to churn out papers to get their degrees (which is beneficial to the universities as they are assessed on how many affiliated publications they output). There's of course plenty of quality PhD-level research but it's also quite obvious that the quality is also affected by the students still essentially learning the ropes.
Academia's, science's and higher education's societial position and "clout" was quite different and smaller in the era of classical liberalism (roughly until WW1), and public funding had a lot smaller role.
It's of course totally arguable that competition leads to more efficiency. However, the academic world doesn't really have a natural market and establishing comparative value of different lines of research, academic institutions or individual researchers is very difficult. And due to this the resource allocation is done largely based on quantity of scholarly outputs (papers and PhD degrees) and salesmanship. These of course are very easy to play, and those who don't engage in the play tend to not survive. And it adds huge overheads.
[1] https://link.springer.com/article/10.1007/s10734-008-9169-6 [2] https://en.m.wikipedia.org/wiki/New_Public_Management
Well, according to the statistics...
The discussion isn't whether or not there's a crisis, it's the reasons for the crisis and what is the root cause? Which I don't believe has been answered as of yet.
Is it because we simply don't actually understand the science and perhaps sometime in the future we will actually understand it? Maybe? But why then do we allege that it's even science at this point?
Is it because of some conspiracy. For example cPTSD ought to be in the DSM, why isn't it? Because it will wipe out half the book? cPTSD better diagnoses the vast majority of personality disorders and such. How about the major breakthrough in cancer and heart disease? China, Ecuador, Mexico, etc have basically confirmed cheap highly effective cure to cancer. Seemingly no limitation on type of cancer, same cure effective for breast, renal, pancreatic. As confirmed by actual cancer organizations published by NIH.
Is it because the STEM movement has basically robbed all the good people from these fields? The people who would have been a superstar in social sciences is now an engineer?
Or my personal favourite, it's political. They write their 'study' with p=0.00005 but conclude they were correct as opposed to a null value. Then the political publications pick it up as if it were p=1