Bizarre. How do people make such big, splashy findings that can mess with people’s sense of optimism about science and innovation, without doing the simplest types of checks on their data and methodology.
There should be a real incentive/compensation for reviewing properly and real consequences if a paper gets retracted for reasons that should have been caught in review.
In this case it's fortunate that it did get found out in the end.
* is the treatment of existing work semi-thorough (even experts don’t know everything) and fair?
* are the claims novel w.r.t the existing work? If not, provide a reference to someone who has already done it.
* can you understand the experiments?
* do the experiments and their results lead to the conclusions claimed as novel?
* does the writing inhibit understanding of the technical content?
No peer review I have ever seen or done would catch anything but the most egregious bug of this nature.
I have definitely done that with benchmarks / profiles.
It’s probably even easier when the incentives encourage “the find”.
Will they own up to it and retract their broken paper that's eroding people's confidence in funding science at the highest levels? This has been an incredibly widely read and influential paper already.
I have my doubts that the authors will accept that their paper is bogus.
Particularly because the lead author landed a faculty position at a good institution based exclusively on this junk paper.
2nd. One of the ways we discover problems with data is by plotting. When the plot library has a bug that hides a problem, well shit.
3rd. They did check their own findings multiple ways. Mistakes happen. The biggest critics of scientific mistakes are often those that have never done science themselves. Its easy, and its a cheap play.
kind of like the difference between "trusting science" and "trusting THE science" if I had to hazard a guess
presumably he doesn't mean as opposed to "traditional ways of knowing"