>
given the context I assumed that claims of scientific fraud would involve evidence.The other commenter did provide some evidence (albeit anectodal) of obvious fraud. In the context of non-replicable studies, it can be as high as 2/3rds are shown to be non-reproducable in some domains. This includes some of the highest quality journals and institutions. There's quite a bit of evidence of this. Now I'm not claiming that is all fraud, but I am claiming we don't know how much of the problem is fraud.
>evidence of absence would be enormously difficult to prove
I disagree. I'm taking the point that it's not hard to prove, but rather the incentives are aligned against it. If data and methods are truly open-sourced, much of this can be proven. But there is little status or money to be gained from reproducing work, leading to a situation where we have a system where it is rarely done.
>yet science has managed to produce massive and visible benefits
I don't disagree, but I think you're are misinterpreting my position. I said the system is flawed, not that it's worthless. Put differently, it still produces good science but it's hard to separate the good science from the bad. The signal-to-noise ratio is potentially low, and the rate of publication exacerbates this. I would argue that boosting the signal-to-noise ratio by weeding out bad science could produce much more benefit because less time is wasted chasing trails of bad science.
>What strategy would you pursue
You keep framing this as some impossible scenario. I'm saying it only appears that way within the context of the current incentive struture. So my goal would be to create a change to that incentive structure. This is off the top of my head, so don't take it too seriously but here's what I would try:
1) We need to change the way academics are measured. Currently, it's a "publish or perish" paradigm that puts a premium on novelty and essentially disregards most everything else. There is also a glut of PhDs; far too many given the number of tenured academic position. This puts an enormous amount of pressure on a PhD to produce publications and this can manifest itself in pushing bad science. There's two things we could do to help this:
1.(a) Provide an incentive where reproducing research is valuable. Currently, researchers don't want to spend time reproducing studies because there's very little upside. They won't generally get published so it doesn't help their career. I would say we need to start giving more status to reproduction studies. Probably not as much as more "novel" studies, but there should be an avenue to publish them. Maybe a two-tiered journal structure where we have *Nature* and *Nature:reproductions*, where the latter is used to publish works that re-create the studies in the first. Metrics like citeScore could be weighted so that the novel studies give you a higher metric than the reproductions, but you still get some credit for either.
1.(b) Provide a pathway to also publish "non surprising" studies. In the current structure, research is more prioritized if it produces surprising results. Thus, if a researcher produces work that is unsurprising, they tend to just throw it out and try something else. This can lead to wasted effort when another researcher ends up doing the same thing. We should create an avenue to also publish "unsurprising" results. This has already started to get applied in some domains, like economics.
2. Right now, publications can hide their data. I think there should be a cultural shift where the highest quality journals make sharing the data and code a mandatory requirement. This ties directly in to facilitating the replicability as well as allowing others to look at your methods and see if there's any wonkiness going on (whether fraud or otherwise). Some of the issues of fraud that we've seen are because researchers couldn't hide their data because it came from a third party (eg, Dan Ariely and Francesca Gino were outted because their data was owned by an insurance company that was willing provide insight into the raw data).
3. Journals should set aside a portion of fees to fund replication. As it stands, journals get free labor from referees. I would argue this should be balanced by having some money set aside to fund a process aimed at replicating some of the studies in journal. It's wild to me that we have a system where a referee is expected to work for free and also to pay for a subscription to the journal they are helping.
There's other ideas about improving the referee'ing of the articles but this is already long enough for a forum post.