PIs may not be able to raise money for replication studies. Some of this is a consequence of guidelines for federal funds to prevent waste on duplication of effort.
There’s no replication crisis for academics because they have a meatspace social network of academics; they go to conferences together and know each other. You can just ignore a paper if you know the author is an idiot.
If medical studies are faked, is it a problem? Presumable regulatory agencies are using these studies or something, right? Looks like the FDA and NSF need to fund some more replication studies.
Many papers aren't even "looks legit but doesn't replicate", they are self-evidently wrong just from reading them or the associated data. Just peer review done properly would solve that, but the problem is widespread because there's no real incentive to rock the boat. Once a bad technique or invalid approach gets through to publication a few times it becomes a new standard within the field because it lets people publish more papers.
There are only two ways to fix that, as far as I can see:
1. Have highly technical, scientifically trained and skeptical politicians who police academia and ensure research money is well spent.
2. Stop governments funding research in the first place.
(1) just doesn't seem feasible. There are so many problems there. (2) is feasible. So I suspect you're going to start seeing calls from the right to defund universities over the next ten years or so. The underlying motivation may be that universities are strongholds of the left, but the stated justification will be the high levels of research fraud.
Attempting to fix this within the framework of government would be incredibly distracting for the state, as it'd lead to endless fights and debates. For example, a reasonable person appointed to run the NSF might conclude they should just defund whole subfields of the social sciences, because there's no way to make them scientific at all, but good luck lasting in the job if you do that. A big part of the problem is that scientists are in hoc to a specific political ideology, so they will have powerful friends in Washington who want to see them stick around producing useful propaganda.
Huge funding for replication studies is commonly suggested, but it won't work. I'm basing that view on experience of having read and reported invalid papers across several different fields, spent years following this story and its various ins and outs, and have written extensively about the problems with pseudo-science coming out of official institutions.
Replication is the polite society way to talk about a whole range of problems. People say there's a replication crisis because it sounds a lot better than saying there's a large scale fraud and incompetence crisis. But it's not like there are papers out there that abstractly don't replicate, nobody could have known, and when someone tries and fails then there's suddenly a whole process that's followed to root cause why it failed and fix it. There isn't anything even close to that. What actually happens is that very obvious problems are ignored or kicked into the long grass as long as possible, often indefinitely, and sometimes scientists even argue they're being victimized by those reporting problems.
This article is a reasonable place to start, if you're new to the problem space:
https://fantasticanachronism.com/2020/09/11/whats-wrong-with...
It raises points that are often overlooked due to the "replication crisis" framing, points like:
• Many papers that can be replicated are only replicable because they're of little value (e.g. poor children have worse exam results). We don't want to be flooded with highly replicable science that tells us only what everyone already knew.
• Many papers are replicable but their claims are still invalid because their logic or methodology is wrong. This category covers significant amounts of COVID science, for example.
• It's often unclear exactly what the definition of replicable is.
But zooming out from replicability for a moment, I wrote an article with about some of the more egregious problems here:
https://blog.plan99.net/fake-science-part-i-7e9764571422
We're talking about a scientific culture that can't even stop obviously machine-generated text from being published on a massive scale. No other part of society has problems like this. Even talking about replication seems like a distraction whilst you have papers being published every day that contain obviously Photoshopped images, mathematically impossible numbers and AI generated text, yet nobody cares and getting even one case "resolved" (paper retracted) requires months or years of external pressure.
Think about how big the trust problems are with journalism. How to fix trust in journalism is talked about a lot on the conference circuit, but it's a hard problem. And yet newspapers don't publish garbled gibberish and faked images on a daily basis! That's the scale of the challenge faced with science reform.
And even if you somehow manage to drain that swamp, the first step before full replication is attempted is to get the original data, so the analysis steps can be replicated using the original numbers. Whole swathes of science fail here because academics refuse to reveal their data. This happens even when they've been made to sign a statement agreeing that they'll provide it on request. Again, their employers do nothing, so where's the pressure point? Only massive fiscal punishment could cause culture change here but the NSF has a single institutional goal of dishing out as much money as possible. To quote Alvaro de Menard,
Why is the Replication Markets project funded by the Department of Defense? If you look at the NSF's 2019 Performance Highlights, you'll find items such as "Foster a culture of inclusion through change management efforts" (Status: "Achieved") and "Inform applicants whether their proposals have been declined or recommended for funding in a timely manner" (Status: "Not Achieved"). Pusillanimous reports repeat tired clichés about "training", "transparency", and a "culture of openness" while downplaying the scale of the problem and ignoring the incentives. No serious actions have followed from their recommendations.
It's not that they're trying and failing—they appear to be completely oblivious. We're talking about an organization with an 8 billion dollar budget that is responsible for a huge part of social science funding, and they can't manage to inform people that their grant was declined! These are the people we must depend on to fix everything.