I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.
Currently, a significant proportion of research results in various fields cannot be reproduced. This essentially means that a lot of work turns out to be flawed, leading to wasted efforts (you can refer to the 'reproducibility crisis' for more context). Moreover, future research often builds upon this erroneous information, wasting even more resources. As a result, academic journals get cluttered with substandard work, making them increasingly difficult to monitor and comprehend. Additionally, the overall quality of written communication deteriorates as emphasis shifts from the accurate transfer and reproduction of knowledge to the inflated portrayal of novelty.
Now consider a scenario where 50% of all research is dedicated to reproduction. Although this may seem to decelerate progress in the short term, it ensures a more consistent and reliable advancement in the long term. The quality of writing would likely improve to facilitate replication. Furthermore, research methodology would be disseminated more quickly, enhancing overall research effectiveness.
Wouldn't this imply that worthwile results are already being replicated, so the primary cost has been paid and we just need to give some method to disseminate this work and then have it factor into the credibility of science? pre-print < peer reviewed < peer replicated, with the last step having internal rankings depending upon how much it has been replicated?
And it'll also show when someone is building on things but not replicating it, which I guess is an issue in some fields more than others.
Peer replication is completely unfeasible in experimental fields of science. The current process of peer review is alright, people just need to learn that single papers standing by themselves don't mean too much. The "peer replication" happens over time anyway when others use the same tools, samples, techniques on related problems and find results in agreement with earlier papers.
Implementing the code for the simulation and analysis of the data? four months, at most. Running the simulation? almost three years until I had data with good enough resolution for publishing.
For a reductive example, the idea to solve P vs NP is a great one, but I’m not going to do that any time soon!
Even if someone meticulously documents their process, it could still take months to replicate the results.
I'm familiar with lithography/nanofabrication and I know that it is typically the case that a process developed in one clean-room can not be directly applied to a different clean room and instead one has to develop a new process based on what the other results.
Even in the same lab it can often happen that if you come back to a process after a longer time, that things don't work out anymore and quite a bit of troubleshooting ensues (maybe a supplier for some chemical changed and even though it should be the same formula it behaves slightly different).
I previously worked in agricultural research (in the private sector), and we spent YEARS trying to replicate some published research from overseas. And that was research that had previously been successfully replicated, and we even flew in the original scientists and borrowed a number of their PhD students for several months, year after year, to help us try to make it work.
We never did get it to fully replicate in our country. We ended up having to make some pretty extreme changes to the research to get similar (albeit less reliable) results here.
We never did figure out why it worked in one part of the world but not another, since we controlled for every other factor we could think of (including literally importing the original team's lab supplies at great expense, just in case there was some trace contaminant on locally sourced materials).
Doesn't that indicate further research is needed? It sounds fascinating to me. (I know it isn't interesting for the people who couldn't get it working.) It also might indicate that the original research was incomplete in the sense that it might be a fluke due to specific conditions in the original country which isn't universal.
You are almost stressing all the ways we are producing garbage rendered non-reproducible with deficient documentation of processes, changes in supply, and changes in the environment. All three can be minimized through peer replication.
Reproduction is hard, really really fucking hard. Just saying, that means we should replicate before trying to understand, means essentially cutting off the understanding.
And like others have said, if someone wants to build from it they'll depend on that information being correct, if no one can manage to ever build from it then the idea dies.
Also there's a huge difference between, replicate this study on infant response to stimulus, or spider colony behavior, and, replicate this incredibly intricate semiconductor that took years of configuration to correctly produce.
There is plenty of science out there which financially, practically, or ethically simply by definition cannot be replicated. That doesn't mean their results should not be published. If peer review shows that their methods and analysis are sound, there is no reason to doubt the results.
> Where are you going to get the budget to build a second LHC solely for replication?
In cases like this you could simply have a second, independent team time-sharing the LHC and using it to replicate experiments run by the first team. (And vice- versa). It’s not a perfect replication but it’s probably still an improvement over the “just trust me bro” status quo.
> How are you going to replicate a long-term medical cohort study which has been running for thirty years?
Run two independent studies in parallel from the beginning.
> What about a paper describing a one-off astronomical event, like the "Wow!" signal?
There was a ton of effort invested into trying to replicate that observation! Since nobody else ever managed to do so, we can’t draw any conclusions from it.
> What if you research the long-term impact of high-dose radiation exposure during Chernobyl?
That doesn’t preclude replication unless, for some reason, you’re the only researcher researching the long-term impact of high-dose radiation exposure during Chernobyl.
That's not a true replication, and it isn't going to avoid issues like the OPERA experiment measuring neutrinos going faster than the speed of light due to a loose connector. It would not be any different from having the second team just run their own analysis on the data from the first team - at which point the second team can just as well simply validate the first team's analysis like peer review is currently doing.
> Run two independent studies in parallel from the beginning.
So all currently-running long-running research has to be thrown out? What if the two studies find very small differences, are you allowed to publish either of them? Are the two teams allowed to collaborate at all?
> There was a ton of effort invested into trying to replicate that observation! Since nobody else ever managed to do so, we can’t draw any conclusions from it.
You can't "replicate" an observation of a freak astronomical event because you can't trigger a freak astronomical event. At best you can do observations and hope it happens again. We indeed cannot draw any conclusions from it, but that doesn't mean you can't publish papers about it. If replication is mandatory, you would not be allowed to do anything with it at all.
> That doesn’t preclude replication unless, for some reason, you’re the only researcher researching the long-term impact of high-dose radiation exposure during Chernobyl.
It cannot be reproduced because it would be unethical to expose people to near-fatal levels of radiation simply for reproduction. Simply reusing data from the original test subjects isn't a reproduction, after all.
As you already quoted me as saying, it's not a perfect replication. Which is fine! I'm advocating a position of "replicate findings as much as reasonably possible", not a position of "we need to build redundant copies of every multi-billion-dollar research megaproject". My whole point is that this doesn't need to be an absolutist true-or-false sort of thing.
> and it isn't going to avoid issues like the OPERA experiment measuring neutrinos going faster than the speed of light due to a loose connector
Maybe not. I never claimed this would solve every problem in all of science forever.
> So all currently-running long-running research has to be thrown out?
No. I think it's reasonable to propose more rigorous standards for future studies without throwing out every in-progress study that didn't follow those same standards. After all, there are literally centuries of published science that didn't even follow the contemporary standards of peer review, and we haven't thrown any of that out.
> What if the two studies find very small differences, are you allowed to publish either of them?
That's an extremely broad question. You might as well ask, "what does it mean for a finding to be replicated?".
If you and I each independently go out to measure the length of the Golden Gate Bridge in millimeters, there are likely to be very small differences in the result you get and the result I get. There's an expected margin of error here where we can agree that our results are consistent with each other. Sometimes the differences are reasonable and can be explained, and sometimes they can't be explained.
Regardless, I also think it might even be valuable for the studies to be published in some form even if they don't replicate at all; just not necessarily with the imprimatur of some credible or prestigious journal.
> Are the two teams allowed to collaborate at all?
I would suggest keeping the two teams independent at least until they both have results. Afterwards, it might be valuable for the team to collaborate in terms of trying to reconcile their results.
> You can't "replicate" an observation of a freak astronomical event because you can't trigger a freak astronomical event. At best you can do observations and hope it happens again. We indeed cannot draw any conclusions from it, but that doesn't mean you can't publish papers about it. If replication is mandatory, you would not be allowed to do anything with it at all.
Yeah, I guess I'm fine publishing a paper that just says "here's this anomalous observation we had" if it's an especially interesting anomalous observation like that.
> It cannot be reproduced because it would be unethical to expose people to near-fatal levels of radiation simply for reproduction.
Obviously. I think you know that's not what I'm suggesting at all here.
> Simply reusing data from the original test subjects isn't a reproduction, after all.
At this point, I agree we're mostly stuck with whatever data we managed to get 37 years ago. But if a similar incident happened in the future, you could have independent teams collecting redundant sets of data.
NSF grants distribute 8.5 billion dollars a year, which is less than Major League Baseball (and its Congressionally granted monopoly) makes. The US Congress has directed 75 billion dollars in aid to Ukraine to date.
This seems particularly problematic because it is already notoriously hard to get tenure and academia is already notoriously unrewarding to researchers who don't have tenure.