I think it would be better if there were incentives that rewarded quality over quantity. At the moment, my university always says that quality is of the utmost importance, but then threatens to terminate my job if I cannot publish x number of papers in a given year.
From an economic perspective, is this a very desirable situation?
If the research isn't documented well enough to reproduce/verify, then the paper shouldn't pass review in the first place. The NIH could make it a condition of funding that papers are detailed enough to be reproducible.
I am not saying that all this is a desirable situation. It is very unfortunate, and I wish there was an easy solution. My first research paper took 5 major revisions and 6 years to get through peer review. All the reviewers criticized was the wording and my unwillingness to conform to the accepted views in that particular community; I almost lost my job over this, but once the paper was accepted, it won several awards. All of this leads me to believe that peer review is very subjective and prone to error, and I don't have a solution for that.
Given the quality of the social science papers I've read this seems like it would be a good thing IFF the 98% cut were the bottom 98%.
I suspect a lot of "hard" science papers would be caught as well so it's a necessary quality control method
Maybe we could have something like that, where the vigilance department receives a small amount of money paid off from the penalty imposed on the researcher with bad/fraudulent research.
They're incredibly empowered to conduct raids and seize property and other belongings, if it comes down to that and the sum involved is large enough.
But only if that sum involved is not large enough for the owner to afford proper defense lawyers
https://www.irs.gov/about-irs/whistleblower-office-at-a-glan...
The second one is a little less relevant for published research, although it could take a different form when implemented in Academia.
To come back to the malicious part, for many researchers, not publishing the exact way they do things is part of how they protect themselves from people reproducing their work. Some do it for money (they want to start a business from that research), others to avoid competition, others because they believe they own the publicly funded research...
Very often, the thing you are trying to reproduce isn't exactly the same that was published. You have to adapt the instructions to your specific case, which can easily go wrong. Or maybe you did a mistake in following the instructions. Or maybe you mixed the instructions for two different cases, because you didn't fully understand the subtleties of the topic. Or maybe you made a mistake in translating the provided scripts to your institute's computational environment.
That means there are important validation/verification steps left out of the whole process. Sure, it's impossible to give every detail, and naturally there's always a time constraint, but if there's a hypothesis of action it needs to be verified. (Again easier said than done.)
For my masters' research I spent 6 years refining a super niche technique until I was able to reproduce my own work.
This is how we write pen testing reports at work. A pen testing report written that way ~20 years ago is one of the things that got me interested in pen testing. But I apply it to all of my technical writing.
If lack of reproducibility in science is as big a problem as it seems to be, maybe journals should impose a randomized "buddy system" where getting a paper published is conditional on agreeing to repeat at least 2-3 other experiments performed by peers. Have 3 peer researchers/labs repeat the work. If at least 2/3 are successful, publish the paper. If not, the original researchers can revise their instructions once or twice to account for things the peers did differently because the original instructions didn't discuss them.
Hopefully needing to depend on the other organizations for future peer review would be sufficient to keel everyone honest, but maybe throw in a secret "we know this is reproducible" and a secret "we know this is not reproducible" set of instructions every once in awhile and ban organizations from the journal if they fail more than a few of those.
For corner cases that require something truly impractical for even a peer to reproduce independently ("our equipment included the Large Hadron Collider and a neutral particle beam cannon in geosynchronous orbit"), the researchers that want to publish can supervise the peers as the peers reproduce the work using the original equipment.
This would obviously be costly up front, but I think it would be less costly in the big picture than thousands of scientists basing decades of research on a single inaccurate paper.
I also think that forcing different teams to work together might help build a collaborative culture instead of the hostile one that's described elsewhere in this discussion, but maybe I'm overly optimistic.
That already happened in CS, which inherited slow and thorough journals from mathematics. Because peer review was taking too long, people elevated abstracts in conference proceedings to the status of papers. The idea was that you submitted an extended abstract to a conference with limited peer review. After receiving feedback, you would write the actual paper and submit it to a journal for proper peer review. But because the work was already published in conference proceedings, people often didn't bother with the full paper.
In some countries, administrators resisted this and only considered journal papers real publications. Those administrators were universally reviled by the CS community. Over time, most of them budged and started accepting conference papers as merits. And so CS became a field with lower than average standards for peer review.
Most research is useless and pointless, with only a few exceptions. We don't have a way to figure out which topics are the exceptions, so someone has to do the research. It's not worth it (or rather: extremely high financial risk) for companies or individuals to do it, so governments gave to fund it.
At this point, the current amount of fraud does not justify replicating even 1% of studies. We would get less scientific advancement in total. The current situation likely does justify some small investments in shaping incentives.
The problem is that it's hard to reliably capture value from research. A good example is LLM's. If OpenAI, Google, Meta and AWS had been able to build a wall round GTP3.5 Turbo and above models then I expect that they could have captured all the value of the research effort.. as it is I don't think that is/will be the case - it's almost too easy to replicate as Mistral have shown. Note: I'm not saying it's trivial or something, but if you spend a few $million on it you can get close enough, and then spending a few $million more will get you all the way. Also, I am not talking about building a frontier model today (which requires $100millon or so and some difficult skills/organisation) but rather a model in say 3 years time with the frontier performance of todays models.
People generally don't want to do the work of editing and publishing, or lack the academic knowhow to do it. But if that is not an issue, I don't think money will be an issue either.
Here's their motivation in a Nature letter to the editor: https://media.nature.com/original/magazine-assets/d41586-020...
Nature, as well as other top journals, do not publish results that report already published findings. Replication Studies would be the exception. These would provide independent validation of a recent prominent article.
People would think twice before misrepresenting findings in top journals, and good ideas would spread out more quickly.
Not necessarily published as part of the paper, a link to the separate document is fine.
That assessment does not match up with what any practicing scientist thinks is even within the realm of possibility for harm to science.
Reading these conversations is like listening to C-suite execs at big companies talk about what employees are getting away with via work at home policies.
Yes and yes. I'm 6 years past defending my PhD and I have low confidence in being able to reproduce results from papers in my field (computational biophysics).
I was recently at an industry-heavy biophysics conference that ran a speed dating event, and my conversation starter was "what fraction of papers in our field do you trust?". I probably talked to ~20 people, with a median response of ~25%.
Even a tiny amount invested in reproduction studies and accountability would go a long way. Most papers in _computational_ biophysics still don't publish usable code and data.
If this happens before founders/early investors aren't the ones left without a chair when the music stops, it doesn't matter
See: Theranos.
Typically, most companies are answerable to the investors and shareholders. Customers usually don't figure in the equation.
And I am intimately familiar with what researchers “get away with’ while ‘working at home’. As a researcher who tried to reproduce several research papers only to discover the original scientists were wildly exaggerating their claims or cleverly disguising fundamental shortcomings, I can assure the cost is quite high to the scientific community, well in excess of 25% of the annual $48B NIH budget.
I hold a healthy disdain for my fellow scientists. The only way to get them to play by the rules in my view is to have a threat of a research audit hanging over them.
For example, the NIH could identify the top findings from 2024 that need to be reproduced, and seek expressions of interest to reproduce these and/or other important findings identified by applicants. Perhaps, also reach an agreement with top journals to publish replications as a new article type, and link them to the original one, just like they do with comments/news & views.
It would instantly make those publishing super edgy findings much more careful, just in case, and things would become more efficient.
Currently, academic publications are in a bit of market for lemons situation [1], where the seller (authors) have much more information than the buyers (readers, funders).
Time to change that.
wdym? You're on the happy path when reproducing, the cost of the original study includes all the failed attempts.
So, yes, the current situation can safely be assumed to pose at least a 25% cost on science. And "productivity" is the wrong term here. The harm of fraudulent/bad science runs much deeper than productivity
However, it did not have methods, it didn't say how they were not reproducible, as in a figure or an effect etc.
The closest thing to a definition of "reproducible" was a footnote on a table defining it as "sufficient to drive a drug development program," which is not at all the same thing as reproducible.
Which is to say, I'm skeptical of these anecdotes.
And if you've only ever encountered a single opinion piece on the reproducibility problem in biology/pre-clinical research, then I highly recommend you do a targeted keyword search.
> then I highly recommend you do a targeted keyword search.
There's no reason to be insulting, especially when linking to well known studies.
"I replicated X's work" or even "I was unable to replicate X's work" isn't exactly a career maker
Although I guess you could get the few staff scientists at the NIH to handle it.
Some older professors who already have tenure might be willing to help out, if they don't already have much else on their plate