Double-anonymous peer review reduces reviewer bias, finds three-year trial
britishecologicalsociety.org
britishecologicalsociety.org
• It seems OK. Large sample size, fairly simple statistical treatment, clear and (mostly) honest writeup of their results.
• About half of reviewers could correctly guess who the authors were despite the double blinding.
• The motivation was the widespread belief in their field that women are discriminated against by peer review. In reality they found a slight bias in favor of women. (Their abstract and press release says they found no bias, but on p11 they say differently).
• They also found no racism. Review results for people from non-western countries didn't change.
• The difference is all due to a strong positive bias towards researchers from rich countries. This is probably "prestige bias", but can also just be that people go softer on people they know already because the reviewers are mostly western.
• Optional double blinding will therefore be ineffective, it has to be mandatory.
This would be an important step forward.
In the worst case, a PC chair would occasionally ask about it, so it would follow him for years.
They may simply have missed it or failed to realize the relationship before so blindly saying "it's fine I will just ignore that they skipped over my own work" when it is in fact related doesn't seem like the right approach.
If it really is unrelated I doubt anyone feels the need to point to their own research.
In practice, reviewers did sometimes request citations of their own works. As an editor you could reject the whole review report, but you were on deadlines and quotas and you typically had a hard enough time finding reviewers anyway. In my experience, most of the practices that researchers gripe about in MDPI are results of onerous working conditions rather than any kind of laxity in the culture; everyone was regularly encouraged to be serious about scientific integrity and seek out the best reviewers, and practically all of the people I worked with were quite intelligent and concerned about doing a good job, but it was also an unspoken expectation to work after hours, Saturdays, to always be sending an email, etc; exhausted people make mistakes.
Also, I think people are overestimating how much reviewers care who the authors of the papers they are reviewing are, at least in my field.
For one thing, the papers are often written by a student, so writing style isn’t a great giveaway. Also, the advisor probably appreciates a thoughtful rejection of the crappy student paper. The alternative is getting it accepted, followed by a mad dash by the advisor to copy edit it and add experiments so the work isn’t embarrassing. (Tenure clocks are the main counter example of this, but junior professors aren’t exactly in a position to bully the reviewers!)
People could try to block work that would scoop their ongoing paper, but I don’t really see that happening in practice in my field. Double blind doesn’t help with that, of course.
However, it would pretty rapidly tank that reviewer’s reputation, since the other reviewers would see the bad behavior, and they know each others’ identities.
Reproducibility is getting better, but still a huge problem (we can't forget the billions of dollars of funding that went to Alzheimer for a scam paper)
Social workers are just trying to reverse some of the damage companies and governments are doing, but I don't think it will help, even I honestly think they are generally good people doing their best.
Maybe disrupting the ad model that just started recently can help, but I believe social studies have no chance to fight against these things.
That's a bold claim worthy of consideration. To assess it, we need to first ask: have the academic incentives always been "publish or perish"? And the answer is...apparently not, according to “Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States”. This paper details how the current paradigm is a product of political anti-science-funding sentiment in the 1970s.
That tracks with anecdotes from Einstein, Higgs, and Fuller regarding distaste with the modern peer review system.
Unfortunately peer review looks like it's actually a policy issue, because it's tied to government funding, so the change in incentives would have to start there. I don't have any answers but at the same time I think it's worth considering the fact that "publish or perish" is not the natural state of academia, but rather a recent phenomenon that appears to coincide with the reproducibility crisis and a decline in the number of large scientific breakthroughs.
What's your proposed alternative? No expectations of output at all? Academia has changed many times over the centuries, it's not even clear it has a natural state. As something essentially willed into existence by governments and mega-donors, it's not even really describable as natural.
The reproducibility crisis probably isn't new. Maybe public awareness of it is, though I'd say the start of the "crisis" is coming up to its 15th birthday now. But there were people talking about that sort of problem in psychology way back in the 60s and 70s, and examples of bad research claims in other fields that became widely can be found readily when looked for.
It seems like reproducibility and other problems are the natural consequence of flooding universities with research grants. Removing the incentive to publish isn't a fix, because then you'd just be flooding universities with money that'd then just disappear, or more likely, get diverted into political campaigning. The money has to be buying something. Because it doesn't buy actual verified results but only papers asserting such results exist, reproducibility problems are an inevitable consequence.
> it was only in the late twentieth century that peer review came to be seen as a process central to scientific practice. Throughout the nineteenth century and into much of the twentieth, external referee reports were considered an optional part of journal editing or grant making. The idea that refereeing is a requirement for scientific legitimacy seems to have arisen first in the Cold War United States.
You're asking what the alternative is, but government-funded research isn't new at all. Only the peer review requirement is new. The requirement doesn't need to be replaced with anything.
Here is what Peter Higgs has to say about it:
> He doubts a similar breakthrough could be achieved in today's academic culture, because of the expectations on academics to collaborate and keep churning out papers. He said: "It's difficult to imagine how I would ever have enough peace and quiet in the present sort of climate to do what I did in 1964."
IMO rolling back academia to the pre-war period is fine if you accept also rolling back the size of it to that era but that would mean almost all researchers losing their jobs. If that isn't acceptable then there has to be some kind of expectation of quality and quantity of output. Yes in the past there wasn't, but that's because so few people worked in academia that it hardly mattered.
Nowadays proposals to get rid of peer review will run into objections from people who aren't satisfied with the quality of even peer reviewed papers (assholes like me). There are way too many published papers that are completely fraudulent or outright auto-generated for eliminating what few checks exist to be reasonable. After all, peer review is really a very low bar. The average programmer both is given and gives far more rigorous reviews of their work every week, and then on top we have automated testing, static analysis tools, crash reporters etc because human-driven reviews really aren't enough to ensure quality output. Scientific research hasn't gone the same way and that's leading to a wider loss of trust.