Reproducibility project fails to validate dozens of biomedical studies

And since we are talking about science reform, let's start with the much easier and cheaper preregistration [1] which helps massively with publication bias.

[1] https://en.wikipedia.org/wiki/Preregistration_(science)

He advocated (very naively, as it appears today) for science as a human endeavor that has no reason for falsification. His justification was that scientists have nothing to lose from being proved wrong, and, as an example, he gave some University dean who published some works that were shown to be completely wrong in a course of few decades, but still retained his position in a university (because his approach was valid and he never attempted to manipulate the truth, he just made an honest error).

But, the more I think about how did we come to this, in many human activities it is often the case that whoever undertook such activities relied on their own wealth and not being incentivized to commercialize their discoveries. It was the aristocrats or monks or some other occupation that made their life affordable, and boring enough for them to look for challenge in art or science. Once science became professional, it started to be incentivized in the same way any other vocation is: make more of it--be paid more; make more immediately useful things--be paid more.

I don't know if we should return to the lords and monks system :) But I'm also doubtful that we can make good progress by pulling the levers on financial incentives of commercializing science.

Incentives like these exist in basically all areas of work. Perform well and you get "job security, promotion and prestige". Yet somehow there is no decade long ongoing crisis in industry of corporations lying about their products. When these cases happens (obviously they do), corporations and individuals get punished.

How would you reform the system? More funding definitely is not the answer.

Here sample sizes were tiny, which introduced a vast amount of random noise. The fact so many studies where replicated suggests the vast majority of the underlying studies where valid not just the ones they could reproduce.

I think I heard this idea from Freakonomics, but a fix is to propose research to a journal before conducting it and being committed to publication regardless of outcome.

Imagine the guy who got a FAANG job and made it nine weeks in before washing out, informing you how the entire industry doesn’t know how to write code. Maybe they’re right and the industry doesn’t know how to write code! But I want to hear it from the person who actually made a career, not the intern who made it through part of a summer.

It's lack of industry experience. I complained about this is a recent comment here: https://news.ycombinator.com/item?id=43769856

Basically, in SE anyway, the largest number of publications are authored by new graduates.

Think about how clueless the new MSc or PhD graduate is when they join your team: thesebare the majority of authors.

The system is set up to incentivise the wrong thing.

In the first year you are clueless anyway, what a better use of your time than replicating some established result, and then publish your findings. In the experimental world the author will often skip details that are crucial to the success, just because it has become second nature to them. A replication study will reveal these, aka you know you need really sterile room to make this work, or you need to set-up this way your linux distribution.

Of course that's what should be happening, but the incentives aren't pushing in the right direction for it currently.

In order to get a PhD out of it, you'd likely need to do a large number of reproducibility work, specifically going off some hypothesis like "method X has failings due to Y" and then testing it by doing the reproduction or similar. This makes it not low hanging fruit. In addition reproducing work is often very hard and time-consuming as you discover just how much is left out of papers/SI/appendicies.

Scientists have informal trust networks that I’d like to see made explicit. For example, I’d like to see a social media network for scientists where they can PRIVATELY specify trust levels in each other and in specific papers, and subscribe to each others’ trust networks, to get an aggregated private view of how their personal trusted community views specific labs and papers.

I don't think you want to slow down publication (and probably peer review and prestiage journals are useless/obsolete in era of internet); it's already crazy slow.

So let's see: you want people to incentivize two things (1) no false claims in original research (2) to have people try to reproduce claims.

So here's a humble proposal for a funding source (say...the govt): set aside a pot of money specifically for people to try to reproduce research; let this be a valid career path. Your goal should try to be getting research validated by repro before OTHER research starts to build on those premises (avoiding having the whole field go off on wild goose chases like happened w/ Alzheheimer's). And then, when results DON'T repro, blackball the original researchers from funding. (With whatever sort of due process is needed to make this reasonable.)

I think it'd sort things out.

Just open up a comment section for institutional affiliates.

There is an huge amount of pressure to publish publish publish.

So, many researchers prefeer to write very simple things that are probably true or applicative work, which is kind of useful, or publish false/fake results.

But at the same time, I doubt that fields like physics and chemistry had better practices in, say, the 19th century. It would be interesting to conduct a reproducibility project on the empirical studies supporting electromagnetism or thermodynamics. There were probably a lot of crap papers!

Those fields had a backup, which was that studies and theories were interconnected, so that they tended to cross-validate one another. This also meant that individual studies were hot-pluggable. One of them could fail replication and the whole edifice wouldn't suddenly collapse.

My graduate thesis project was never replicated. For one thing, the equipment that I used had been discontinued before I finished, and cost about a million bucks in today's dollars. On the other hand, two labs built similar experiments that were considerably better, made my results obsolete, and enabled further progress. That was a much better use of resources.

I think fixing replication will have to involve fixing more than replication, but thinking about how science progresses as a whole.

Then you perform the experiment exactly* how you said you would based on the pre-registration, and you get to publish your results whether they are positive or negative.

* Changes are allowed, but must be explicitly called out and a valid reason given.

https://en.wikipedia.org/wiki/Preregistration_(science)

IMO, the best way forward would be simply doubling every study with independent researchers (ideally they shouldn't have contact with each other beyond the protocol). That certainly doubles the costs, but it's really just about the only way to catch bad actors early.

One issue is that internal science within a company/lab can move incredibly fast -- assays, protocols, datasets and algorithms change often. People tend to lose track of what data, what parameters, and what code they used to arrive at a particular figure or conclusion. Inevitably, some of those end up being published.

Journals requiring data and code for publication helps, but it's usually just one step at the end of a LONG research process. And as far as I'm aware, no one actually verifies that the code you submitted produces the figures in your paper.

It's a big reason why we started https://GoFigr.io. I think making reproducibility both real-time and automatic is key to make this situation better.

Often famous/more cited studies are not replicable. But if you want to work on similar research problem and publish null/non exciting results, you're up for a fight. Journals want new, fun, exciting results but unfortunately the world doesn't work that way

And then, once you got your PhD, only then you would be expected to publish new, original research.

They are continually prescribed because their actual mechanism doesn't matter, they demonstrably work. That is a matter of statistics, not science.

Anti-science types always point to the same EXTREMELY FEW examples of how science "fails", like Galileo (which had nothing to do with science) and ulcers.

They never seem to point to the much more common examples where people became convinced of something scientifically untrue for decades despite plenty of evidence otherwise. The British recognized a link between citrus and scurvy well before they were even called "Limeys"! They then screwed themselves over by changing some variables (cooking lime juice) and instead turned to a quack ("respected doctor" from a time when most people recognized doctors were worse than the sickness they treated) who insisted on alternative treatment. For about a hundred years, British sailors suffered and died due to one quacks ego.

Phrenology was always, from day one, unscientific. You STILL find morons pushing it's claims, using it to justify their godawful, hateful, and murderous world views.

Ivermectin is a great example, since you can create a "study" in Africa to show Ivermectin cures anything you want, because it is a parasite killer and most people in impoverished areas suffer from parasites, so will improve if they take it. It's entirely unrelated to the illness you claim to treat, but nobody on Facebook will ever understand that, because they tuned out science education decades ago.

How many people have died from alternative medicine quacks pushing outright disproven pseudoscience on people who have been told not to trust scientists by people pushing an agenda?

How much money is made selling sugarpills to idiots who have been told to distrust science, not just "be skeptical of any paper" but outright, scientists are in a conspiracy to lie to you!

In addition to his sub stack, his Twitter is great and very accessible

> The teams were able to replicate the results of less than half of the tested experiments1. That rate is in keeping with that found by other large-scale attempts to reproduce scientific findings. But the latest work is unique in focusing on papers that use specific methods and in examining the research output of a specific country, according to the research teams.

Agreed. Lab technique is a thing. There is a reason for the dark joke that in Physics, theorists are washed up by age 30, but experimentalists aren't even competent until age 40.

For psychology replace "Difficult" with "Pseudo".

To lose that tag, Psychology has to take a step back, do basic research, replicate that research multiple times, think about how to do replicatable new research, and only then start actually letting psychologists do new research to advance science.

Instead of that, unreplicated pseudo-scientific nonsense psychology papers are being used to tell governments how to force us to live our lives.

Both psychology and nutrition are some of the most error prone areas of science. Yet results from both of these are again and again publicized as major finding. "Scientists find ..." headlines have conditioned the population into believing uncritically the outcome of scientific studies. Of course scientists are also heavily incentivized to portray their results as definitive and get publicity among the general population for their research. (By the way, this is surely what contributed to the growth of the "anti-science" movement. You can only read so many obviously fake science papers in the news about how "X is curing/causing cancer" until you realize that there is something deeply wrong.)

All the incentives are terrible. And unless there are major changes scientific credibility will fall even lower.

>Science is "self correcting" because if the "effect" you saw isn't real, nobody will be able to build off your work. Alzheimer's Amyloid research has been really unproductive, which is how we knew it probably wasn't the magic bullet even before it had fraud scandals.

After how many billions wasted? By that metric we do not need airplane safety regulations, as after ramming a hundred planes into the ground the margins become bad and safety should improve.

Science is full of fraudsters and they should not be allowed to operate like this.

It would be worse if the experiments were not even falsifiable, yes.

But it’s pretty damn bad when the conclusion of the original study can never be confirmed when once in a rare min they try.

In your example, it’s the same as someone publishing a paper that disproves Relativity - only for us to find that the author fabricated the data.