The Absurdity of Peer Review: What the Pandemic Revealed
elemental.medium.com
elemental.medium.com
The author begins by discussing the importance given in popular media to whether articles are peer reviewed or not, using somewhat absurd examples of foundational historical papers to stress the point.
The author then makes a logical jump, and claims that "Science worships peer review". However neither the journal Science nor the discipline of the same name are capable of worshipping anything. The author might mean to say that "Scientists worship peer review"? But for such a claim the author provides little evidence, and further it would contradict the later statement that "simple inertia, and ..." are the reasons "we still have peer review".
Subsequently the author tries to demonstrate that peer review is useless by stressing the multitude of problematic papers that did make it through peer review. This, of course, is a standard logical fallacy and again in direct contradiction with the latter statement that "we do not know whether the science we produce is better with or without it" (emphasis mine).
As I mentioned above, it is clear that the author raises an important point and at its root the message it tries to convey is certainly valid. However the article itself adds little to what scientists (and I would wager the informed public) already know. Besides the issues mentions above, I think a further weakness is that it claims to speak for all of science but I might imagine that in a field like mathematics the act of peer review can be more meaningful.
Altogether, then, I am of the opinion that the above issues need to be corrected in a substantial revision before I can accept the article for publication in your journal.
Sincerely,
Now extrapolating this outside of niche science and into research that is underpinning policy decisions (or tenure and promotion decisions), of course one would expect it to undergo some kind of review. And if something had not, and was just a preprint being circulated, it's a pretty important consideration that its not yet part of established canon.
Certainly I agree there are lots of flaws with peer review, that a system that originally evolved to build out the body of accepted knowledge in a field had now been formalized and used for other purposes, and that it can be gamed. But it don't find it credible to say that we just don't need it, and that we shouldn't distinguish between new results and better scrutinized ones.
Compare Humphries' statement: "None of the classic papers on DNA, nor Einstein’s four in his miracle year, were peer reviewed.
with this quote from Wikipedia at https://en.wikipedia.org/wiki/Scholarly_peer_review (emphasis mine):
> Peer review became a touchstone of the scientific method, but until the end of the 19th century was often performed directly by an editor-in-chief or editorial committee. Editors of scientific journals at that time made publication decisions without seeking outside input, i.e. an external panel of reviewers, giving established authors latitude in their journalistic discretion. For example, Albert Einstein's four revolutionary Annus Mirabilis papers in the 1905 issue of Annalen der Physik were peer-reviewed by the journal's editor-in-chief, Max Planck, and its co-editor, Wilhelm Wien, both future Nobel prize winners and together experts on the topics of these papers.
The latter quote supports your expectation that Einstein's work was indeed scrutinized by his peers.
I'll use that distinction between internal and external peer review to analyze this quote by Humphries: "And the media have dealt with this explosion by consistently pointing out when research has not yet been peer reviewed. Presumably they do this to warn the reader that the research lacks the safeguards that peer review brings."
This means the media is reporting that a given preprint has had neither internal peer review by an editor, nor external peer review.
Also, it's hard to tell whether peer review actually improves paper quality by comparing published / unpublished papers in a world with peer review, where everyone is writing with the knowledge they'll be intensely scrutinized. Without some sort of detailed review process -- even one that's potentially deeply flawed -- researchers would have fewer incentives to be careful.
‘It is ridiculous to think covid-19 could have originated in a lab’
‘Covid-19 spreads through touching surfaces’
‘Hydroxychloroquine cures covid-19’
‘Hydroxychloroquine does not cure covid-19’
‘Any opinions that go against this popular scientific stance are dangerous and should be blocked’
In fact, it took people about one day from publication to start pointing them out with proof - and these people were (supposedly) less qualified than the peer reviewers.
It is true that, with properly faked data, a reviewer cannot be expected to catch it. Nor can p-hacking generally be discovered from an incomplete description.
But quite a bit of faked and even p-hacked data is less-than-competently done in a way that a reviewer can and should be able to catch;
And observational data or meta-study are much easier to verify, since no experiment was done by the researcher either.
How would you solve this problem?
This isn't self-evident. In fact, it's conceivable that peer review could actually lower the quality of published research by delaying paradigm shifting work from seeing the light of day.
It's a sign that something might well hold up, but it's not definitive. I think people know this about their own field, and then immediately forget it when dealing with another field.
Have there been any peer-reviewed studies that unambiguously admit the possibility that Covid-19 may have originated in a lab? Although I think it's something that needs serious consideration, the alleged lack of Medical Reliable Sources (basically medical journal articles) is being used aggressively by a small group of editors on Wikipedia to enforce the line that a lab leak is "conspiracy theory".
Alternatives to peer review are seen everywhere these days: open review and discussion and even the open source software model.
A scientist does some work, writes it up in rough form, puts data files online, and tells some friends/colleagues about it. After some feedback and corrections, the scientist puts the writeup online and notifies discussion forums. Then, anyone interested can look it over and provide feedback of all sorts: good, bad, and ugly.
The scientist then incrementally upgrades the paper, including edits and rewording and donated graphics, etc., along the lines of open source software and pull requests. I'm not claiming that many eyes make all bugs shallow, but if someone points out that your algorithm omits the first line of each data table, or your evidence of warming only measured temperature during the day while comparing to temperatures measured day and night, it's true or not regardless of the credentials of the person making the claim.
The paper doesn't have to be fossilized but can be versioned, because "paper" doesn't mean paper anymore, and your institutional worth (as opposed to scientific worth) can be based on citations of all versions rather than the prestige of the journal.
You might fear that authors will be overwhelmed by mostly low-quality feedback, but most papers are read by so few people that this isn't much of a risk, in my opinion. The biggest risk is deliberate lies, but we already have a reproducibility crisis, and only reproducibility--not the filters of authority--can ultimately solve it.
The solution is to make peer-review better, rather than remove it entirely. Imagine that you can choose a set of people who you trust on any particular topic, and rank them with how much you trust them. Then you, and those people, can all rank any paper (or in general any URL) you read, and each of you will have a feed of incoming papers (or URLs) on any topic computed by a function that you control, in your personal reader app, which crawls your network of trusted people, and the people they trust, and so on, and aggregates all rankings and reviews. After about 7 degrees of hops across that network, you'll cover most of the world.
The key new idea here is that peer review will no longer try to be a single, objective, determination of truth. Each person can have their own slightly different feed, computed by slightly different criteria, using slightly different sets of trust metrics of people over the community, all competing for other people's trust, by writing good reviews, and good papers, in order to gain reputation and influence within the distributed network.
This won't be a single app. Rather, it'll be a distributed use of the WWW. Anyone can host their ratings at any web URL, on any website, and subscribe to the ratings from anyone else, at any other URL, at any other website. We can make this all realtime using the web extensions defined by https://braid.org, so that you can have a continuously-updating feed.
In effect, we'll be building a distributed, subjective hacker news (or equivalently, a distributed subjective reddit). We don't need to fight over objectivity anymore. Instead, we'll all be entitled to our own view of the web, and be able to share our views with others. And we'll find that we are best off ourselves when we import the best views from the best people we know, and that way we will see the world through each other's eyes, and even be able to — with the flip of a dial — see the world through our enemies' eyes.
Please reach out if you are interested in building anything similar! I am toomim@gmail.com, and am coordinating a group to make this happen.
Peer review is not what the pubic and journalists think it is.
An example:
The Physical Review journals publish experimental high energy physics papers from large collaborations without review, because there is no point to it. Anyone competent to review these papers is already on the list of 400 authors.
Science is strongly specialized. Results are largely disseminated by preprint. You know if a preprint makes sense if you’re working in the field. You probably know better than whomever the editor can get to agree to review the thing.
By the time something gets published, a year has gone by and research is already building on work reported in the preprint.
Of course it requires that there isn't any hidden agenda or that a specific result of the review is dictated.
Before I ever did any reviewing, I always felt like some conferences really are like a lottery on whether or not you get accepted. You can submit the same paper and get nothing but strong accepts or strong rejects maybe even with the same reviewers.
When reviewing papers, I think I understood more about why. When reviewing you can see other reviewers' comments when you're done with your review, as well as their level of expertise. With some really really niche areas, some people simply aren't qualified to be reviewing, specify their expertise in an area as higher than it is, and give a really half-assed review. Part of that is also because we need to review a whole bunch of papers and can't dedicate hours to each one.
The only time I've found this process to work properly is with really good papers or really shit papers. There, everybody tends to agree. For average papers, it's not that you have a mixture of weak accepts and weak rejects and it comes down to the wire, but rather it's a mixture of strong accepts and strong rejects, with comments that praise/roast a paper that you're not even sure is the same one you read, and you doubt your own understanding of the topic. There is usually very little structure and objectivity built into the system, so mediocre science presented really well is much more likely to do well. That's why it's so random.
I never thought of this, and it is one of those shifts in habits and uses that go unremarked (until, of course, someone points them out like this piece did).
Perhaps due to the industrialization of research after the middle of the XX century?
Now that we don't really need to publish in journals, since anything can be made available on the internet, the filter of credibility can also be changed - for instance to something like being cited approvingly by other reputable researchers.