While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.
Sociology/Psych/Economics are almost all junk. Their conclusions may or may not be correct.
Medical studies are mostly junk. There's way too much financial incentive to show marginal improvement. Theraflu and anti-depressents come to mind. Both show a small effect in studies and launched billion dollar businesses.
Hard science stuff tends to be pretty good. Mostly just outright fraud and they usually end up getting caught.
Also a lot of stuff is tough to classify. Are epidemiology or climatology "hard" sciences?
Or how many studies are useless, period? It's like publishing a memoir to Amazon. You can now say "author" on your resume, or when you're introduced or at cocktail parties but nobody finds any value in what you have to say. You can also use ChatGPT because people might not notice.
They've always known so there hasn't actually been any new information from which to spur action. In the academic circles I've run in there has always been a strong mistrust of reported results and procedures based on past difficulties with internal efforts replicating results. Basically a right of passage for a grad student to be tasked with replicating work from an impossible paper.
You might think that publishing about the replication crisis itself would be great for your career, but perhaps not. Maybe the incentives to be able to bullshit your way to a professorship are so great that no one wants to rock the boat.
Our whole economy is fueled by people bullshitting each other.
Proving something is false is less valued than proving it right. It's silly because if we valued the quality of science it should be the exact opposite.
Many papers aren't even "looks legit but doesn't replicate", they are self-evidently wrong just from reading them or the associated data. Just peer review done properly would solve that, but the problem is widespread because there's no real incentive to rock the boat. Once a bad technique or invalid approach gets through to publication a few times it becomes a new standard within the field because it lets people publish more papers.
There are only two ways to fix that, as far as I can see:
1. Have highly technical, scientifically trained and skeptical politicians who police academia and ensure research money is well spent.
2. Stop governments funding research in the first place.
(1) just doesn't seem feasible. There are so many problems there. (2) is feasible. So I suspect you're going to start seeing calls from the right to defund universities over the next ten years or so. The underlying motivation may be that universities are strongholds of the left, but the stated justification will be the high levels of research fraud.
Attempting to fix this within the framework of government would be incredibly distracting for the state, as it'd lead to endless fights and debates. For example, a reasonable person appointed to run the NSF might conclude they should just defund whole subfields of the social sciences, because there's no way to make them scientific at all, but good luck lasting in the job if you do that. A big part of the problem is that scientists are in hoc to a specific political ideology, so they will have powerful friends in Washington who want to see them stick around producing useful propaganda.
Huge funding for replication studies is commonly suggested, but it won't work. I'm basing that view on experience of having read and reported invalid papers across several different fields, spent years following this story and its various ins and outs, and have written extensively about the problems with pseudo-science coming out of official institutions.
Replication is the polite society way to talk about a whole range of problems. People say there's a replication crisis because it sounds a lot better than saying there's a large scale fraud and incompetence crisis. But it's not like there are papers out there that abstractly don't replicate, nobody could have known, and when someone tries and fails then there's suddenly a whole process that's followed to root cause why it failed and fix it. There isn't anything even close to that. What actually happens is that very obvious problems are ignored or kicked into the long grass as long as possible, often indefinitely, and sometimes scientists even argue they're being victimized by those reporting problems.
This article is a reasonable place to start, if you're new to the problem space:
https://fantasticanachronism.com/2020/09/11/whats-wrong-with...
It raises points that are often overlooked due to the "replication crisis" framing, points like:
• Many papers that can be replicated are only replicable because they're of little value (e.g. poor children have worse exam results). We don't want to be flooded with highly replicable science that tells us only what everyone already knew.
• Many papers are replicable but their claims are still invalid because their logic or methodology is wrong. This category covers significant amounts of COVID science, for example.
• It's often unclear exactly what the definition of replicable is.
But zooming out from replicability for a moment, I wrote an article with about some of the more egregious problems here:
https://blog.plan99.net/fake-science-part-i-7e9764571422
We're talking about a scientific culture that can't even stop obviously machine-generated text from being published on a massive scale. No other part of society has problems like this. Even talking about replication seems like a distraction whilst you have papers being published every day that contain obviously Photoshopped images, mathematically impossible numbers and AI generated text, yet nobody cares and getting even one case "resolved" (paper retracted) requires months or years of external pressure.
Think about how big the trust problems are with journalism. How to fix trust in journalism is talked about a lot on the conference circuit, but it's a hard problem. And yet newspapers don't publish garbled gibberish and faked images on a daily basis! That's the scale of the challenge faced with science reform.
And even if you somehow manage to drain that swamp, the first step before full replication is attempted is to get the original data, so the analysis steps can be replicated using the original numbers. Whole swathes of science fail here because academics refuse to reveal their data. This happens even when they've been made to sign a statement agreeing that they'll provide it on request. Again, their employers do nothing, so where's the pressure point? Only massive fiscal punishment could cause culture change here but the NSF has a single institutional goal of dishing out as much money as possible. To quote Alvaro de Menard,
Why is the Replication Markets project funded by the Department of Defense? If you look at the NSF's 2019 Performance Highlights, you'll find items such as "Foster a culture of inclusion through change management efforts" (Status: "Achieved") and "Inform applicants whether their proposals have been declined or recommended for funding in a timely manner" (Status: "Not Achieved"). Pusillanimous reports repeat tired clichés about "training", "transparency", and a "culture of openness" while downplaying the scale of the problem and ignoring the incentives. No serious actions have followed from their recommendations.
It's not that they're trying and failing—they appear to be completely oblivious. We're talking about an organization with an 8 billion dollar budget that is responsible for a huge part of social science funding, and they can't manage to inform people that their grant was declined! These are the people we must depend on to fix everything.
There’s no replication crisis for academics because they have a meatspace social network of academics; they go to conferences together and know each other. You can just ignore a paper if you know the author is an idiot.
If medical studies are faked, is it a problem? Presumable regulatory agencies are using these studies or something, right? Looks like the FDA and NSF need to fund some more replication studies.
What can they do? It's an incredibly hard problem to solve. It's like asking why the buisness community has done nothing to adress the housing crisis.
Large scale culture changes, or the entire structure of the way science is conducted and funded, would be the only solutions
* working groups "red teams" whose whole job it is to find weaknesses in papers published at the university
* post mortems after finding papers with serious flaws exposing the problem and coming up with constructive corrective actions
* funding / forcing researchers to devote a certain amount of time to replicating significant results
* working groups of experts in statistics and study design available to consult
* systems to register studies, methodologies, data sets, etc. with the aim of eliminating error and preventing post-hoc fishing expeditions for results
The whole-ass purpose of a university is seeking knowledge. They are fully capable of doing a better job of it but they don't because what they actually focus on are things like fundraising, professional sports teams, constructing attractive buildings, and advancing ranking.
Most universities would be better off just firing and not replacing 90% of their administration.
That's because universities are in the business of teaching. Apart from a few rare exceptions, universities don't have the money to hire redundant people. Instead of hiring many experts in the same topic, they prefer hiring a wider range of expertise, in order to provide better learning opportunities for the students.
Statisticians can find faults in many studies. Universities even employ statisticians to act as consultants for research in other departments, as that raises the quality of the institution's publications.
Make peer review public. Weight replication studies more. Make conducted peer reviews and replications an important metric for tenure and grants. Publish data and code alongside papers.
It doesn't help. It just means outsiders get to read bad peer reviews and bad code, but the right people within the system don't actually care. There's no way for the public to hold anyone to account, science is a law unto itself because politicians are generally not scientifically minded. They can be easily bullied with technobabble. The few that aren't susceptible to this don't tend to rise to the top.
There's literally thousands of different things they could do.
But why do anything when the real business of academia is in the tax-free hedge funds they operate and the government-subsidized international student gravy train? There's no short-term incentive to change anything.
Dude. The scientific community created the replication crisis.
We are not ignoring it, I promise you. The reaction is nuanced by no one who matters is ignoring it.
This is putting it very mildly.