It's all incredibly depressing. I really feel for all the junior researchers who end up wasting their time, and often derailing their careers because they followed a path based on other people's academic fraud.
No, the better approach here is to just shift the incentives. Start funding replication. Once we see labs and career paths that specialize in replication / knowledge consolidation, the whole system will shift for the better. Bibliometrics and hiring committees will start to pay attention and then exploratory researchers will start to pay attention and the system will start to work a little bit better.
I’m not convinced this fixes anything. Even when a result is genuine, it’s very easy to fail to replicate it. We all know this from software development: it’s a lot easier to say “couldn’t reproduce” about a genuine bug than it is to track down the precise context in which the bug actually manifests. So if you get rewarded for failing to replicate a result, all the fraudsters will do that. If you get funded only when you actually replicate the result, the fraudsters will pretend to replicate the result.
To the extent that is true, then it is itself evidence supporting the proposition that not-yet-replicated results should be regarded as provisional.
If a study claims to prove something, it should be repeatedly provable or it's a) fraud or b) not proven solidly enough.
I think replication is a key component of a functional research.
So reward either? This seems pretty obvious.
A bigger issue is that... 3-sigmas is really a weak signal in a high cardinality state-space, which is basically everything above physics of small numbers of elementary particles, and it is the elementary physicists that go for higher. This is what is feeding the replication crisis: Weak signals from very poorly sampled studies.
The meta issue is that we as a society need to start accepting that some things will take longer and require more investment to achieve results. Do fewer studies per unit grant, but do the three-sigma ones only to justify a real experiment/study, not as an acceptance criteria for "discovery!".
I'm not even going to touch politicalization, since I really have no idea what to do about it without a worse cure than the disease.
Yes!
> Start funding replication
No :( That won't solve the problem, it'd just change shape. Funding people who produce fraudulent or incompetent papers won't stop being a problem if you ask them to produce a slightly different kind of paper. You just get fraudulent replications, fraudulent claims of failure to replicate, fraudulent rebuttals to claims of failure to replicate and so on.
The other thing this would do is sharply shift the distribution of papers towards those that are guaranteed to replicate, either because they prove things that are trivially true or because they prove things about simulated worlds that may or may not be connected to reality. Both categories of paper are already way too popular, nobody needs more.
Unfortunately, the incentives fix for this problem is so blindly simple it's also difficult to bring up in polite company, because lots of people have an allergic reaction to the implications. Look at places where science is being done at scale without replication crises or widespread fraud, and notice what's different about the incentives. Then ensure science is always done with those incentives.
I suspect educational institutions don't want to be seen as organizations that sue their own researchers. Same with paper publishers. And same for grant issuers.
Downstream researchers? Do they have the time, the money? ability to show direct damages whose recompense would be worth all the effort?
Students affected perhaps could, but only if the effect was very direct and they had the resources. And desire to be known as someone who sues their professor.
The damage is usually so diffuse. There is no one party with all the reasons to sue.
I have no idea what the process would be for criminal prosecution, but the diffuse impact may be an inhibitory factor there too.
Obviously that doesn't apply in setting like drug development where standards do exist, as defined by the best currently available treatments. But if someone is working on something like psychological studies where replicability is the exception rather than the rule, or on exotic tech where only one experimental facility might exist, or on substances or effects that exist only under weird conditions, it's not always that easy (or that safe) to accuse them of lying. Even when you're pretty sure they are.
But there's also the old and established parts, like statistics, parts of the experimental setup, methodology, reporting data accurately (or at all). This is plenty enough to have objective standards for.
A lot of fraud is not in making up experimental results, but instead misreporting the data and drawing unsupported conclusions.
And there are plenty of fields where you have different interpretations of the same data (see the entire field of economics, also plenty in physics and other fields). Should the people who espoused ether theory be sued for fraud? It'll be a huge mess because doing research is by definition doing something unprecedented.
Even once you are over that hump you will have quite a few years going from one short-term grant to another, with your ability to get funding being dependent on your previous work. If that has been stymied because you were basing it off dodgy research of other people it could take a long time to build up the kind of record where you get to diversify and some any kind of academic security of freedom.
That said, putting all your eggs in one basket is often necessary to get anywhere with that basket of research.