> "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it."
"Is there any idea so outlandish that it won’t be published in a Critical/PoMo/Identity/‘Theory’ journal?" tweeted the Harvard psychologist Steven Pinker.
"Three intrepid academics," wrote Yascha Mounk, an author and lecturer on government at Harvard, "just perpetrated a giant version of the Sokal Hoax, placing … fake papers in major academic journals. Call it Sokal Squared. The result is hilarious and delightful. It also showcases a serious problem with big parts of academia."
From what Pinker wrote, who knows if he even read it?
7⁄20 or 7⁄17, depending on how you want to count is concerning.
You could try working without any faith in your colleagues of course, but you are probably better of switching workplace if that sounds like reasonable behaviour to you.
You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science.
We have to stop pretending like the peer review process is some kind of magical filter of true science, or that it was ever intended to be as such. That notion leads to the faulty idea that if it's published, it's true. Junk science gets published, even if it's not disingenuous, and that's probably true no matter how great you make the peer review system.
Explicit fraud isn't that common, since it does have risks to the submitter that far outweigh the (limited) benefits of getting an extra publication on your CV.
I've got a stack of rejected papers that beg to differ.
> Isn't the point that they serve as a basic filter for obvious noise?
Yeah, and obvious noise is usually rejected. But sometimes it is not so obvious what the noise is, and that's why the scientific process is bigger than the peer review system.
> Isn't the point that they serve as a basic filter for obvious noise?
Totally, there is a lot to fix about it. But if you really want to put your effort into changing the system, I don't think you should focus on peer review. Destroying the business model of journals is where I would start.
The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed. But we always approach reviews with a charitable lens, even when reviewing for extremely low-acceptance outlets.
The process that detects the bad research comes afterwards, as it should.
It clearly failed to do that in this case.
A system that prevents what the authors did here would necessarily reject a huge amount of useful research.
They can't verify that the data collected actually was collected without literally watching over the shoulder of the researcher while the researcher was collecting it, which is obviously not feasible. A researcher actively trying to deceive the reviewer with fake data but with a good method for collecting that data and analysis of the data that would be correct if the data were real is not something that should be caught by a reviewer. That should be caught by someone reproducing the research after publication.
I would submit that "fake research" is, in fact, bad research.
Instead I rely on the fact that the community will attempt to integrate these ideas after publication and arrive at the conclusion that the paper was garbage.
The Wikipedia article on peer review says "Peer review...occasionally detects fraud, but is not designed to do so."[1]
The article cites a Nature article that says "Peer review is not currently designed to detect deception, nor does it guarantee the validity of research findings. It should, however, identify flaws in the design, presentation, analysis and interpretation of science and provide prompt, detailed, constructive criticism to improve research."
[1] https://en.wikipedia.org/wiki/Scholarly_peer_review
[2] https://www.nature.com/nature/peerreview/debate/nature05007....