The rise and fall of peer review
experimentalhistory.substack.com
experimentalhistory.substack.com
It seems like the author's beef is with journals, rather than peer-review. If we did away with "peer review" today, journals would still have to operate the same way -- they'd still have many more submissions than they have room for, so a team of people (ideally, peers) would need to, uh, review those submissions according to some criteria. We can discuss whether the criteria should be adjusted, but I don't see how journals survive without gatekeepers.
So, fine, he wants to do away with journals. Without a description of an alternative system, it sounds like the best researchers would just...upload their stuff to Arxiv and hope that someone reads it? Again, I'm not saying there is no alternative, but because he spends all his time arguing against "peer review", he spends no time discussing alternatives to journals that would solve more problems than they create.
He addresses the question "can we fix peer review instead of replacing it" by discussing ways that fixes have failed in the real world. So what makes him think that a replacement would be easier? The "burn it all down and rebuild it according to my preferences" approach also doesn't have a great track record!
And the certainty with which he states his conclusions gives me a sense that this is not someone who's super open to feedback.
> The issue is too systemic at this point
Ah, but there are also systemic issues that prevent Congress from doing the things you'd like them to do. For instance, Congress is vulnerable to money. Journals have money they can use to lobby. Are there other players with money and incentive to lobby for the "burn it all down" side? Which mountain of systemic issues is easier to climb? Sounds like an empirical question.
Solving the problem by either approach is going to be really really complicated and full of compromises you won't like, and anyone who talks like their solution is super simple and obvious should be read with skepticism.
So my supervisor does some digging, and guess who reviewer #5 just so happened to be...
Frack academia
Write to the PC chair and the steering committee of the conference. Those kinds of actions do have consequences. At a minimum you should be able to request a conflict-of-interest review of the situation.
2- Outside of engineering (or at least in the Chemistry/Bio field where I came from), conference submissions are abstracts not papers. There may sometimes be special issues where speakers are invited to submit a paper after the fact. But "paper" here means drafting the full manuscript and submitting it to the publisher when you're ready to be published in a future issue. So no steering committee, just an Editor-in-Chief who conducts research in the field. In fact referencing conference abstracts was always frowned upon because they never contained enough information to be properly scrutinized.
Speaking of, this whole "reputable papers only happen with conferences" phenomenon was unheard of to me until I joined an engineering company
This is true:
> From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church. Scientific journals appeared in the 1600s, but they operated more like magazines or newsletters, and their processes of picking articles ranged from “we print whatever we get” to “the editor asks his friend what he thinks” to “the whole society votes.” Sometimes journals couldn’t get enough papers to publish, so editors had to go around begging their friends to submit manuscripts, or fill the space themselves. Scientific publishing remained a hodgepodge for centuries.
This is true as well:
> That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals. That funding turned into a deluge of papers, and journals that previously struggled to fill their pages now struggled to pick which articles to print. Reviewing papers before publication, which was “quite rare” until the 1960s, became much more common. Then it became universal.
Finally, this is also true:
> Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been running for six decades.
That is what I keep hearing from my academic acquaintances as well - When a measure becomes a target, it ceases to be a good measure. And this is what happened with publish or perish.
Let's not forget that Katalin Karikó, a giant in the mRNA field, was demoted by UPenn in 1997 because they considered her work unproductive. But mRNA was precisely the topic that required a multi-decade commitment.
Fortunately she was stubborn like hell. God know how many other people give up after being degraded like that.
It all comes down to this ^ for me. Advances take decades from start to finish. Institutions/organziations (including funders) need to start not just funding innovation (1-5 year cycles), but investing, really committing, to 20-100 year plans, at minimum.
My biggest fear is that hands are thrown up in the air, and science turns into yet another "content" producer (there are signs, to me, that this is exactly where we are headed). Can't produce science^D content fast enough? Skip peer-review. Community of experts getting in your way? Ignore them. We need to get science into tik-tok, because think of the reach for STEM! Really, do we need science to equate to 15-second dopamine hits?
None of this to say that there aren't problems with peer review, but, at least in the field I'm in, peer review is critical, useful, and an important aspect of becoming a scientist. If anything, learning to peer-review teaches one how to critically evaluate others work. Are there people who aren't taught how to Peer review? Sure. Does this mean that we should stop peer-review? The logic doesn't follow.
I've written the backstories for 30 of my papers here: https://jeffhuang.com/struggle_for_each_paper/
Looking at my past experiences for all my papers, I can only conclude that the process is more nuanced than can be described in a tagline like "peer review is broken" or "publishing is random", so I'm interested in hearing everyone else's experiences too.
A few journals are slowly moving toward a better system where publication happens first, then the reviews are published. As a scientist, I would certainly prefer to read the reviews without them blocking/delaying publication.
However, the real problem with peer review is its role in awarding grants, not in publishing papers. I have no idea how to fix that problem without introducing other problems. Too much incentive to game the system.
I would say it's more like a club where everyone knows each other. If you are authoring a paper you often know who the potential referees are (those that published most in the field), and you politely ask the editor to exclude referees that are in open conflict with you. Editors usually abide and you get a reasonable review of your paper.
I agree about most other issues raised about peer review in general and related incentives, but the issues raised here are not some that I recognize.
My understanding is that physics has used arXiv much longer and more consistently than biology has used bioRxiv, so communication suffers less from the impact of delayed (not just blocked) publication.
Physics is a smaller research community than biology, so it is not surprising that club effects have more pronounced. Some less funded sub-fields of biology definitely have that feel (e.g., ecology and evolution), but anything remotely biomedical seems to suffer from the rando reviewer effect much more.
In my experience, the historic "squishiness" of biology and the rapid growth of NIH funding (and hence graduate students mills) leads to a larger population of less rigorous thinkers than in physics. We don't have math requirements to filter those folks out.
"We can't say anything about this, so it's clearly a total failure." lol Seems like the author wants this both ways.
Also, I don't know why they think replication would be more likely if we didn't have peer reviewers at all. Feels like a larger jump from "your study won't be reviewed" to "your study must be replicated" than from "your study is being reviewed" to "your study is being replicated".
Do you mean as far back as the early days < 2000 or did this happen well after they were established (> 2015). If the latter, how much actually changed?
As the author points out, no one has done the control. We don’t know how much worse the scientific literature would be without peer review, and we don’t no how many important insights are lost because of poor reviews. But as someone who has done a lot of peer review (and has had papers rejected with uninformed reviews), it is difficult to state how much worse things could be. (But perhaps that is an overstatement, as very few papers do not find their way into some journal).
The scientific literature today is overwhelming. Peer review helps make it less so.
all of human history prior to the last 50 years? The term "peer review" didn't even exist until the 60s
https://scalar.usc.edu/works/the-art-of-academic-peer-review...
Perhaps part of the argument has to do with the specific form of peer review. In the 18th and 19th centuries, much more science was communicated via correspondence and books. But I suspect there was still a review process, just one that was less democratic and more dependent on reputation.
I'm sure we can improve it, but if science since the 1960s is a failure, than almost everything in human history is a failure.
So yes. Practically everything in human history is a failure.
Because there's no proof. The only thing we know for certain, is that a lot of money is being spent on it, and that some of the mechanisms that are supposed to work can't work (i.e. in the case where people look at and trust preprints anyway).
There will definitely be anecdotes of peer review having caught things (though it'll be harder to show that that prevented harm). But likewise, there are also many, many anecdotes that it has been actively harmful.
No proven upside, limited anecdotal upside, and proven downside. If peer review as a publication gatekeeper hadn't already been a thing, we wouldn't introduce it — that would be considered unscientific. Sounds like something that's ripe for reconsideration.
But you might ask yourself, why do so many scientists waste so much of their time doing peer review. Perhaps they are ignorant, or easily duped, or perhaps they believe that peer review is a valuable use of their time.
The question is not why scientists spend time doing peer review. The question is: why do academics waste time bending over backwards to adapt to the whims of a particular reviewer even beyond when they feel the input is useful? (And the answer is above: the career credentials.)
Again, I'm not agains peer review by itself. But it should not be a gatekeeper to publication, nor have this much leverage on people's careers (a binary decision for every article).
> or perhaps they believe that peer review is a valuable use of their time.
Or perhaps they believe that if they don’t do that work then bad papers will be published under the authoritative rubber stamp of the journals and that they will not need to spend the same amount of time down the road finding out if/how the fascinating study is wrong after having paid 40$ for the pdf.
When I was in academia, I did peer review initially because it was fun to be on the other side for once. And later mostly as a favor to the editors who asked me.
Actual research includes keeping up to date with peers progress.
If you were doing research but not regularly reading new publications anyway, either you were working on a less-fundamental-research side of things, or, without even realising it, you were exploring directions already explored or losing time by not profiting from a new useful approach.
It is just incorrect to say that the time used to read carefully a peer paper is taken over the time of doing research: if you were not peer-reviewing this paper, you maybe would have been reading carefully the same paper in order to keep up to date with your peers.
In fact, one can lose its time peer-reviewing, but its their own fault: in general, you just need to target the review for articles that are useful for you to read anyway.
That seems to be the point that the anti-peer review people are missing.
You then need to assess exactly what is this percentage of still being published despite having things wrong in it. If this percentage is small, peer-reviewing is still more profitable then no peer-reviewing at all (in which 100% of the bad papers are published).
Have you ever worked in research on a topic a bit trendy or fascinating for the general population? Because the world is full of people convinced that they are the new Einstein
It's a bar to clear. It helps to increase the signal to noise ratio. It's not perfect - bad articles get published, good articles get rejected. But imperfection is part of life.
One thing is that knowing your paper is going to be peer reviewed might discourage a lot of cranks from trying to publish. Also, I believe editors have the ability to outright reject papers before they even go out to peer review. This cuts down on completely irrelevant and terrible articles.
The one thing has nothing to do with the other.
I have seen many "peer-reviewed" articles regarding Covid in 2020-2022 that were no better than blog posts. It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias.
Peer review is worth no more than the "peers" who review it, their objectivity, personal investment in the claim, and the level of effort they put into the review. For example, if the "peers" are a group of people who are all seeking to confirm their prior beliefs about a controversial topic, the peer review process is almost completely useless as a signal of quality. Papers that confirm groupthink are routinely shunted to the high-status members of the group for like-minded "review" and rubber-stamp approval. Papers that challenge groupthink are almost invariably nit-picked and/or dismissed entirely. Many never even make it to peer review, because the journal editor rejects the submission.
Throw in issues like reviewer bias toward "established" authors and fields, as well as lack of blinding and selection bias amongst the reviewer pool, and you have a recipe for a formalized system of groupthink.
I believe I was explicit in saying it wasn’t perfect.
What alternative is there that doesn’t have something resembling peer review, isn’t curated by third parties, doesn’t suffer from confirmation bias/groupthink, and doesn’t leave it up to authors to sift through 100s of papers of unknown quality a day to learn about what’s going on in the field?
Certainly, it doesn't come close to the level of credibility that you've attributed to it (i.e. papers without peer review are somehow "blog posts").
> What alternative is there that doesn’t have something resembling peer review, isn’t curated by third parties, doesn’t suffer from confirmation bias/groupthink, and doesn’t leave it up to authors to sift through 100s of papers of unknown quality a day to learn about what’s going on in the field?
There's no free lunch. Nobody serious in any scientific field relies on peer review to determine paper quality. You have to read the paper in front of you and use your experience, which means you must read not just "100s of papers", but literally THOUSANDS of papers, in order to have any reasonable chance of being a credible evaluator.
If you don't have this experience, you can't know the quality of a piece of scientific literature. That's just how it is.
This, of course, when the scientists themselves are honest, acting in good faith and eager to fix their mistakes as Stanovich and Toplak were.
In openly political fields like sociology and whateverstudies, you get "researchers" blatantly manufacturing their conclusions in scale construction.
As one example of some of the gems of scientific integrity in these fields is the finding that conservatives are higher in hostile sexism. Will any liberal check this? Of course not, because it conforms to their expectations that their political opponents are bad. Even if they are honest, if they just glance at a paper they won't question it, and it won't feel weird to read in a quick report on the research.
How was the sausage made? In at least one study, the entire hostile sexism scale was two items long, and one item was, not kidding, "feminists are making entirely reasonable demands of me." And in the greatest surprise ever people dislike their political opponents. Bam, conservatives bad, because we're measuring their attitudes towards politically partisan activists, not the fairer sex.
(The question also presumes that feminists' demands are reasonable. If the activist set are insane, a low score on the question would be an indicator of sanity, not misogyny or simple political partisanship. As an example, Simone de Beauvoir: "No, we don't believe that any woman should have this choice. No woman should be authorized to stay at home to raise children. Society should be totally different. Women should not have that choice, precisely because if there is such a choice, too many women will make that one. It is a way of forcing women in a certain direction.")
There's also the famous case of Bob Altemeyer's Right-wing Authoritarianism scale which is explicitly constructed so that leftists will score lower on it than rightists. Cue decades of results that authoritarianism is more prevalent on the right. Quelle surprise.
(Disclosure: I volunteer for one attempt at such an alternative, https://plaudit.pub.)
How is that clear? There are plenty of papers out there that contain glaring errors and got accepted anyway, if you regularly read publications you should know this better than anyone.
Also you confirm one of their criticisms and contradict your own argument that peer review helps improve scientific literature. What do you base that on, apart from belief?
The argument perhaps implies that spending less money would not lower scores, but as put here does not imply anything about raising scores.
OR if there needs to be better incentives around finding nothing out. Like, given the goal is getting published, it feels like there should be better incentives for papers around not finding anything in an interesting way, or hypothesis testing and discovering your hypothesis was wrong.
I think the correct take away is spending on education is only very loosely coupled to outcomes once some very low threshold is met.
only in your imagination does it imply that.
I'm not a researcher, but I've interviewed a lot of researchers in the context of data sharing. I'd go out on a limb and attribute this reluctance not to lack of any dataset whatsoever in most cases, but to a feeling of ownership over your "IP", and an instinct to protect it. In a niche scientific domain, it's not totally paranoid to worry that you could be giving your data to the competition. I heard this again and again, though mostly it was everyone else who did it, not the interview subject...
There's also just the work involved in getting the data in shape to send to people, which can be a pain in the butt.
The "experts" very rarely take enough time to actually understand the paper and question its assumptions, and often aren't even qualified to do so.
I had one article on the Xerox Star, way back in the day, that passed peer review for TOOIS with just some reasonable suggestions to cite related work.
However, this one: https://papers.ssrn.com/sol3/results.cfm (which was cited in an amicus brief to SCOTUS for the CLS Bank case)
was rejected by CACM, by an "expert" review at Microsoft Research who admitted he knew nothing about patent law. Microsoft is a big believer in software patents, so he's hardly unbiased.
His comments said at one point, because he talked to a patent lawyer "the patent law has changed!" while clearly not knowing that the laws on obviousness had not changed at all.
* There are less great discoveries because there are less great and easy things to discover
* Time has filtered the scientific studies of the previous centuries
* You don't look at the efficiency of a filter (peer reviewing) by the amount of shit that pass through, but by the rating between this amount and the amount of shit that doesn't pass through
* A lot of researchers don't want to release their raw data not because they invented their data but because they want to keep the monopoly for them
* Peer reviewing has failures and could be improved, but it is the best and quickest way to judge a scientific work (with credentials let's be honest)
Reduce the social, political, professional, and academic cost of "being wrong", learn to have a little more empathy and suddenly negative reviews will hold more meaning once again. People will learn to learn instead of learn to defend. (Of course, defense is a skill worth learning as well. Please don't take what I'm saying as this-or-that.)
You simply cannot do away with peer review! What you can eliminate is the control strictures that exist today which punish failure unfairly. This is true at all levels, from the highest institution right on down to that conversation you had with a colleague over coffee yesterday.
Where peer review gets a bit more shady and political is in the federal funding system, where the outcome is not a publication, but delivery of thousands to millions of dollars in federal funding. That's where politics really comes into play, although nobody in any field wants to talk much about it publicly.
Or you could just consult an Ouija board.
Or, peer as in someone of equivalent actual knowledge and experience? AI isn't there yet, but I imagine a machine learning model trained on published papers in a given field would be able to quickly spot a lot of simple errors and/or poorly supported conclusions.
> That debunked theory about vaccines causing autism comes from a peer-reviewed paper in one of the most prestigious journals in the world, and it stayed there for twelve years before it was retracted. How many kids haven’t gotten their shots because one rotten paper made it through peer review and got stamped with the scientific seal of approval?
The point is not to prevent it from getting out there in the first place — that is already possible, you can just upload a PDF to some website. Doing that for a paper arguing 1+1=3 wouldn't be any more accepted if peer review's role would be limited.
The latter has its place, certainly, but before the 1980s that place was industry-funded private research institutions, the most famous and well-known being Bell Labs. These institutions were funded by groups of corporations who in return got first dibs on their patentable work, and the researchers and engineers in turn got good salaries and didn't have any teaching responsibilities.
After Bayh-Dole passed in the 1980s, universities were allowed to grant exclusive licenses to their patentable research to private entities, and this resulted in the rise of the corporate univesity model - people running around hiding their data until they got the patent, people fudging results to make their novel drug look good so their startup would be bought by some pharma corporation, administrations being overrun by pharmaceutical executives who only cared about universities generating a steady stream of patents, and forget about quality-of-teaching.
It's been a complete disaster, and not only that, the private research world has largely been defunded as corporations realized that, with the aid of politicians, they could just outsource basic R&D to universities, using grad students as cheap underpaid lab labor on projects where they're little more than glorified lab techs, and not really doing any original research.
In addition, the whole system is increasingly authoritarian and censorship is rife - any academic who calls for the elimination of exclusive licensing and the adoption of open-source drug discovery will soon find themselves without a job. It's similar in many ways to the Lysenko era in Soviet science, just run by corporate authoritarians instead of Stalinists. Avoid if possible.
Yes, well, how shall I put it? Not everyone who attempts to publish a paper is Albert fucking Einstein.
https://docs.google.com/presentation/d/1SRWbyA2uCnda1JwAZ3Ji...
Worships peer reviewed science journals and believes that AI and technology will save us
Really makes you think.
Peer review is worst system possible, except for all the others that have been tried.
I'm aware that many subfields have good solutions.
But, from the outside as a non-researcher, nobody has conceived of a truly effective "whole of science" narrative so far.
"Financial Conflicts of Interests and the End of Evidence-Based Medicine"
https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
- Bureaucratization. The modern university model is basically the DMV. It functions to sustain the lifestyle of the administrators in charge and to prevent them from losing power, money, status. And no one enjoys it, not the employees, not the customers, etc. and yet who can change it for the better? It can and does only get worse, more bureaucratic, more soul sucking. Ask the post-docs.
- Feminization of academia and science. This is related to the bureaucratization. Process, safety, paperwork, meetings and community consensus are paramount. All becomes politics. Everyone must agree. Only small questions can be answered. You need permission for everything. Anything truly novel is considered a threat to the scientific community. Modern day "scientism" and other beliefs are the replacement to christianity in the west, and women are more religious (look it up), and academe is the modern day church. All dissent or inquiry is squashed.
- Denial of Great Man Theory. Great men invented modernity, and of course our modern industrial state requires many layers of managers due to the huge complexity, so sizes of labs, colleges, corporations, assembly lines, supply chains have exploded, and no one person can manage it all in their mind.
The false belief is thus: modern industrial society requires the managerial class to function, therefore the managerial class invented it, and it requires the managerial class to progress. Therefore great man theory, or the idea of the innovative genius, is supposedly debunked.
And yet you see nearly all major advances come from tech bros like Elon Musk et al who are trailblazers, ignore complaints of the bureacracy, etc.
- Could the university system have produced facebook, spacex, tesla, microsoft, apple? You can go on and on.
Please don't interpret this as an attack on women or femininity. Society requires everyone for it to function properly, and everyone has a positive contribution to make, but we have to be able to iterate and change when we realize certain modes of endeavor simply do.not.work.
I don't want to give specific examples of research topics to avoid poisoning the well, but could you argue with a straight face that an academic that wanted to do research into some topics that could be likely to yield certain types of politically incorrect conclusions wouldn't face extreme difficulty getting funding or extreme risk to their career?
I think it's easy to see how much scientific research on certain topics could be stuck within a narrow range of opinion because people are more concerned with what gets funding or doesn't get them shunned.
>Denial of Great Men...[bad stuff]
>Please don't interpret this as an attack on women or femininity
Bureaucratization is the consequence of wanting to control that taxes are "properly" used.
> - Feminization of academia and science.
That's abolutely moronic viewpoint if you ever worked in science
> - Denial of Great Man Theory
All scientific works, even by genius, lie on the works of others, predecessors or colleagues.
> And yet you see nearly all major advances come from tech bros like Elon Musk et al who are trailblazers, ignore complaints of the bureacracy
lol what scientific progress Elon Musk did except giving material for the sociology of internet trolls ?
> - Could the university system have produced facebook, spacex, tesla, microsoft, apple?
You're so close to understand that university and industry are not the same and have not the same goals
Most researchers I knew had little interest in patents, so non-corporate funded papers got published without patent considerations. When it came up, it was a before-you-publish-think-about-it topic.