Scientific progress despite irreproducibility: A seeming paradox
pnas.org
pnas.org
Scientific results can be strengthened by replication, but that's just one thing, and if that's all you do, then you end up with a science that does nothing but generate independent factoids.
On the other hand, robust science tends to look at a particular phenomenon from many different angles, and manages to connect multiple results together into a framework that can survive retracting individual studies without collapsing. This is how electromagnetism developed.
Sciences that are in the factoid phase are not necessarily junk. Discovery of a psychological "effect" is perfectly scientific and interesting. But some sciences have barely progressed beyond the factoid phase. And if those sciences are also plagued by irreproducibility, then they may embrace scientific methodology without producing a useful scientific knowledge base.
In cases where there is significant differences in the maturity between theory and experiment, you either end up with theorists playing games with math, making up tons of unconstrained theories (string theory, for instance), or in the other direction you end up with experimentalists measuring anything and everything they can imagine, half of which is probably not theoretically enlightening, waiting for theorists to constrain their space of possible experiments to potentially fruitful paths.
Obviously this is harder for fields that don’t have models as robust as we do in physics, but I’d guess the same phenomenon happens everywhere.
So, where is the progress coming from?
Your argument that "there must be a robust and explanatory theory that goes along with experimental results" helps make sense of TFA, which otherwise presents as a paradox, or a ruse to excuse shoddy practices.
The presence of a viable theory is a useful heuristic for sorting useful experimental practice from less useful. At least, I imagine it is. How can we know for sure?
Has anyone published a meta analysis that checks for this?
And has that meta analysis been independently reproduced?
Nonetheless, reproducibility is a really nice property. You have to be incredibly careful with your theory if it can't be justified.
Scientific results are what makes science universal; since they can be reproduced everywhere by anyone.
A variety of causes have been proposed, such as abuse / ignorance of statistics, bad experimental design, and even outright chicanery.
In your view? Reproducibility is essential to scientific progress because it defines what science is. It's the backbone of science. Hypothesis,testing/experiments. It's what separates science from math, arts and pseudosciences like social "sciences".
> I'm sure if you went back through the literature on something like electromagnetism, you would find results that fail to replicate, yet the theory of electromagnetism if applied properly is remarkably robust.
How would it be "robust" if it couldn't be replicated? How would the theory of electromagnetism be robust if it failed to produce replicable results?
> Scientific results can be strengthened by replication
No. It can only be a "scientific result" if it can be replicated/tested. It's definitional.
> Discovery of a psychological "effect" is perfectly scientific and interesting
No it isn't if it can't be test/replicated/reproduced. It's not science. It's something else altogether.
> And if those sciences are also plagued by irreproducibility, then they may embrace scientific methodology without producing a useful scientific knowledge base.
There is no "scientific methodology" without testability/replication/reproducibility.
All science is based on hypothesis-testing.
I read that in the century after Newton, the French Academy offered a prize for evidence of the failure of Newton's laws. They gave out the prize dozens of times, yet Newtonian physics kept getting stronger and stronger. Eventually they stopped giving out the prize. Many of the contradictions apparently had to do with the lunar orbit, which was poorly understood.
Deals with some of your points.
I’ve been a reviewer for a journal and we’d always recieve a good chunk of marketing whitepapers dressed up in just enough jargon to be published.
That seems like it's an important part of this issue. If we know that there are massive problems with irreproducibility and we just have to go on an assumption that there has been "rapid progress" then it seems like the most obvious solution to the "paradox" is to question that progress as well.
This is similar to how in the 90s everything was labeled "healthy and fat free!" which we know resulted in food filled with sugar, then making the claim "the paradox of healthy food despite high sugar". It might be worth calling into question the claims of healthiness given the fundamental conditions that produce health aren't present.
Certainly there are places that everyone will agree we've seen progress, but all of these places seem to be where scientific progress is closely tied to commercial application.
One of the best examples that I think everyone will agree has been remarkable has been the storage capacity of batteries. While much of this progress surely starts in the labs, the true measure of progress isn't in papers it's commercial applications. It frankly doesn't matter at all if the academic research behind battery technology was good or not, because we know these batteries work and are in fact smaller. If you cellphone weighed 10 lbs and ran out of energy in 45 minutes you wouldn't care either way what the research said.
However the answer to this "paradox" isn't just that the market is some force for testing what is real. Anyone who has worked long enough in machine learning knows a fair amount of bullshit not only exists in papers but in products as well.
A better explanation for me is that we live in an age of unprecedented economic and high energy intensity activity. You generate 160,000 TWh of power you're going to see a lot of wonderful things that look like progress. Some real, some illusions, but assuming that science is really the basis for all of this is a fairly large, and unchecked in this article, assumption. Lots of the scientific progress we've made in the last few decades has happened outside of an academic research lab, and assuming "science" is the cause might be a bit naive.
But I would claim that especially in biology, though this is less than ideal for writing up near little explanations of reality, it is still extremely useful for understanding what's going on. Scientific papers aren't meant to be ever-lasting truth, like a textbook. They are communications amongst specialists about "look here something cool happened that may be useful to you too." It is only through lots of work that a phenomenon can be established as widespread in biology, and sharing information before doing 10 years of work helps accelerate everything.
This is why I roll my eyes at complaints like that one about "foundational cancer research papers not being reproducible." It was written by scientists in industry who wanted to take a new paper and develop an entire drug program around something particularly novel and surprising. Sure, that would be ideal if it worked, but the scientific literature is a lot more than a catalog of ideas ready for commercialization. Scientists in practice understand the limitations of taking a journal paper as gospel. They always try to get something working in their own hands before basing a lot of research off another paper.
I've never looked at the historical literature from, say, 100 years ago, but I suspect that it's quite the same as today in all regards.
There's a ton of great ideas that never get the attention they deserve, a bunch of ideas that get way too much attention due to fashion or due to influencers that chose the wrong path. But who could we trust to chart a better path? Just like startups, we must accept a high failure rate when exploring the unknown.
A lot of published results only pass peer review because you essentially trust the authors not to have made up their numbers.
Whatever you might have to say about research in psychology, it's also the field primarily turning the microscope on itself. This is part of a tradition in the field -- modern meta-analysis has its origins there.
I'm less familiar with physics but there's a lot of problems with reproducibility in many fields.
Indeed, which is why I specifically singled out cell biology. It's the part of medicine that's closest to physics. Oncology, pharmacology etc. study humans/whole bodies, so not only are the experiments more expensive to run and much more noisy, there's also all kinds of ethical issues. Most of these aren't there, or are at least reduced, when dealing with just cells (I didn't expect "cell cultures take a long time to grow" and "the chemicals used are non-reproducible" issues that sibling comments pointed out, so I guess not quite physics)
For physics, consider how difficult it is to settle the controversies around something like the EmDrive, even among experimenters with solid reputations and impeccable professionalism.
"[...] It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that..."
It is irreproducibility in a few fields. Most notably, nutrition, psychology, and economics.
These are all fields which study humans. The study of humans is far more fraught with bias and ideology. Humans are also independent decision making agents and behave in a way that atoms do not.
From one point of view that just proves your point more, from another, less. If placebo effect is tenuous then all else equal that’s a good sign for the rest of medicine… but in fact lots of medicine is tenuous. The landmark “Most Published Research Findings Are False” was looking at medical findings iirc.
We can't predict human behavior because our science is simply not advanced enough. I think once we have as strong an understanding of biology as we do something like say, physics or mathematics, we'll find it significantly easier to predict human behavior.
Or put another way, I don't think we can't predict human behavior because humans are "special", but because we're actually kind dumb on a cosmic scale.
Another post here made the point that it is necessary to guide empiricism with strong theoretical frameworks. Those strong theoretical frameworks are missing in psychology/neuroscience, because it is a very very new field tacking something very very complicated. We have weak theories, and thus empirical findings may fail to replicate for differences we do not measure (season? time of day? menstruation? obesity? coffee? as examples of things that might not be measured, or is not practical to include into statistical models due to df) and so on. But this is not to say that there has not been progress. There has been lots of progress, and theories are becoming better, ever so slowly. but we build on converging methods spanning basic neuroscience of individual neurons or local networks to MRI studies of macro phenomenology.
In the end, I think of myself more as a cartographer or explorer, much like those that set sail across the sea knowing little about what will be found, BUT DOCUMENTING IT in their naval logs and reports, so that one day those observations can be put together and build the world map. That is why open neuroscience is critical to our field.
Of course there's no dishonesty. And to be bad would require what you do to be science in the first place. From my perspective (physicist), "science" is applied too broadly. What you do isn't useless or unimportant. I just struggle to square it with "science" in any meaningful sense. That's probably not a popular opinion, but whatever: I stand by it.
it's like protesting overmedication of type two diabetes patients by pointing out that surgery isn't very effective. I agree, it isn't, but we know why the patient has type 2 and how he could cure himself!
Maybe.
And maybe a bunch of those people were self-medicating with alcohol, nicotine, etc. and now are actually getting real treatment.
Don't get me wrong. I think we overly medicate people--especially school children. I think ADHD is hideously overdiagnosed.
However, I've also seen the flip side. People who were completely disconnected from reality who suddenly are back to their "normal" selves with just a small amount of medication.
The number of correct papers grows logarithmically with the number of papers published.
I don't think anyone would argue that more than 50% of scientific publications or findings are flawed.
Back to the “gradient” analogy, in this case, papers have a feedback mechanism to suppress less useful papers. Science is not a random flurry of results mashed together.
I have definitely seen some highly cited and influential papers overstay their welcome and hold the field back.
That's not to say they didn't propel the field forward when they were initially published, just that their overbearing influence causes us to stick to a paradigm that is no longer producing results.
"Science progresses one funeral at a time" and all that.
I didn't see a lot of evidence of outright fraud, but a lot of mundane and irrelevant research ("This algorithm happens to work well on this one particular test dataset well") presented as if it were a breakthrough that was important to the field.
"Publish or perish" is a helluva drug.
The question is: what else could we incentivize that would be better and would still allow us to scale science?
One of the hardest things about scaling human activity is figuring out what your goal functions should be. I don’t think raw publication count is a great metric.
The one I personally always used was engineering usefulness, but that only works for those areas that are close to application. It’s an increasingly useless metric as you get far from application, but the stuff that is far from application is essential to our large scale understanding and to future applicable research.
Marvin Minsky suggested a fairly reasonable solution to this problem based on his experience in the DARPA program during the 60s and 70s: give smart/accomplished people funding and then stand back and let them work out the rest.
A decent amount of the funding will be wasted, but it's still better than the current system of jumping through hoops and appealing to the biases of the old guard in order to get grants.