Is Peer Review a Good Idea? (2020)
journals.uchicago.edu
journals.uchicago.edu
This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Programme Committees are relieved to find a negative review, however unfair or off-kilter it is, because more rejected papers will decrease the acceptance ratio of the conference, hence make it appear more competitive.
Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.
Over the years, I've drifted to the biological side of bioinformatics methods. The papers I write are similar to CS papers, but the projects are bigger, and I no longer submit papers to CS conferences. My typical experience with peer review is accept after 1-3 rounds of revisions. Hence I get to see more of the positive side, where the reviewers act as editors and their suggestions improve the paper.
Of course, it's possible to get a CS-like experience with journals by being ambitious. If you try your luck by submitting the paper to prestigious journals it's unlikely to get in, your typical outcome is a desk rejection or rejection after a single round of reviews. Then you get to see more of the negative side of peer review. But I'm just not interested in playing that game.
Also hard agree on publishing reviews. It seems to be the only viable modification of peer review that we have.
My latest experience with peer review was just bad. No matter the conclusion they drew, there was just a clear lack of quality and understanding in their thoughts and effort. And this is not infrequent.
More generally, such a modification has knock on effects that probably warrant some thought on what we do with our publication practices. Published reviews will probably require more work from reviewers who are already time poor. Better reviews, hopefully, but fewer of them and fewer papers?
Counting a good review as a citable publication might be worth considering, as essentially mini idiosyncratic literature reviews. Perhaps in combination with Registered Reports[1], where the proposal is published and reviewed before the study/work is done, in which case the reviewer is closer to symbiotic co-publisher.
Either way, it seems that for both authors and reviewers, who are effectively the same people and both rely on peer-review as a guarantee of the value of their work, that the actual work of peer review needs to be taken more seriously and not conceived of as some sort of feudal aristocratic gentlemen’s duty.
~~~~~~~~
In some fields like AI knowing the exact authors is also quite common. Because the vast majority of researchers are employed by just a handful of big labs, the reviewers (and sometimes organizers) with experience in the subfield are employed by exactly the same lab(s). So they already know each other and each other's research anyway. Add to that that citations and prior work usually give away the set of authors too.
One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.
What's the difference between posting stuff on arxiv or uploading a pdf to your blog?
> Big labs have big publicity and thousands will read their papers.
Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
There isn't. My point is that the papers aren't blind because they've been read before. Publishing onto your blog would have the same result. I'll note that this isn't common practice in other research communities.
> Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
I don't see how this could even happen. Let's even say you are a no name lab and get accepted, you're in a list of hundreds or even a thousand other papers. There's publicity, yes, but it isn't massive.
Let me put it this way. If you are a grad student at MIT or Stanford you will get hundreds of twitter followers relatively quickly. These followers are going to see you posting links to your papers. You have a larger network effect just by being at a larger and more prestigious school. That is the main advantage of prestigious schools and I don't think there's anything necessarily wrong with that.
But we do have to admit that there are factors that are highly important here beyond merit based. Being at MIT or Stanford does not mean your paper has more merit than a no name school or lab. That no name lab will not have the same reach, which results in fewer citations (our main metric of relevancy), and fewer researchers will be able to expand upon the work.
The major problem here is that we're pretending that this doesn't happen. I don't know how to solve this issue or even know if it can be solved. But it is really dumb to go around saying everything is merit based and not acknowledge the other factors at play here. It is pretending that there's a stronger signal in the noise than there is.
That misses the whole point of anonymous reviews, or even anonymous voting.
You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved.
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
If you pick a journal to publish your paper, which means you explicitly want the paper's editors to go through each and every single line of text you wrote to poke holes, but in the end once those holes are picked you complain that the journal you picked is not the right one for your flawless paper and that they all suck and their problem is that they don't understand your genius, what does this say about you and your work?
When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Your comment reads a whole lot like "the fox and the grapes".
This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well.
> Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc.
Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't.
> When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career.
[1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.
For example, in fields where the peer-review is done primarily by academics are there better reviews during academic breaks vs. say, exam season? Does this translate to more accepted work because the reviewer can spend more time digesting the content?
Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.
Yes, but there is unfortunately a wide variability in the knowledge of reviewers. It's not uncommon to have reviewers directly contradict each other about the shortcomings of the paper. E.g.,
Reviewer 1: "Section A needs to be condensed, this level of background is unnecessary. Readers will find this as common knowledge."
Reviewer 2: "Section A should be expanded as all readers may not be knowledgeable in this area."
Particularly for grad students, this can make the process feel arbitrary and the feedback of limited value. There also can be an inordinate amount of stylistic review comments (e.g., "however" should be changed to "nonetheless"). These often come across as the reviewer feels the need to critique something because they don't have sustentative critiques. Good critiques strengthen a paper, bad feedback seems to do nothing but check the process boxes.
I actually agree with the OP that one potential solution is to encourage reviewers to decline reviewing papers that they don't have the requisite knowledge to provide good feedback. Unfortunately, this decision is often only offered based on reading the title and abstract.
The meta-reviewer doesn't just send the reviews to the authors with no commentary. They write a summary of the reviews with a recommendation about what to do next. If there are contradictions and no consensus is reached, it is their job to decide which concerns the authors need to respond to in order to get it published. They aren't bound by the reviews, they can disregard one if they feel like it is unfair, although sometimes the rules say they must get another reviewer to write a 4th review.
The open discussions not only benefit the peer review process but also act as context and learning material for others.
Never heard this argument, but having been under many double blind peer reviews in the last 10 years, I can sign this.
Interestingly, in my last paper for PLOS One, one of two reviewers chose to lift anonymity and specifically his comments were on-focus, supportive and substantial.
As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.
Of course, in practice, this has the problem that it is already hard to get reviewers, because nobody has time anymore. Removing the anonymity of reviewers leads to the judge becoming open to judgement themselves, and many would not like that. Furthermore, there might be severe repercussions if the reviewed is more powerful than the reviewer, especially in authoritarian societies. But maybe one should just exclude these societies from peer review, and let them do their own thing.
I think peer review has to become a market place. Let everyone choose for themselves which paper they would like to review, and there is both positive and negative credit for both reviewers and authors. The pool of reviewers of a paper shares 25% of the credit that the authors of the paper get. This way you are incentivised to review even outlier papers, because if they are successful you get a substantial amount of credit for them. How the credit is distributed among the reviewers should also incorporate a time factor, so that reviewers rushing to a paper that is already successful don't get nearly as much credit as a lone early reviewer.
I suppose there can be some other layer of reviewer meta-review to account for this, much like the role 'acceptance rate' has come to have for journals and conferences. But, following Goodhart's law, even that has come to be gamed now that a meaning - a proxy for journal prestige or quality - has been placed on it.
I quite like the fact that whatever someone says on the internet now, the first comment is usually the best-argued counterpoint. It seems like a dynamic that could be brought to science quite well. I may not have the time to parse an entire paper in a field unfamiliar to me, but if I come across it and the top comment is 'there's a math error in table 6 that throws paragraph 4 into doubt', I can certainly verify that for myself. That sort of 'peer' review would make science much more accessible to the interested layperson.
Hence, they will sometimes try to make sure you don't publish or at least get downgraded to sister journals, such as Nature Communications.
Another problem are editorial decisions. Great work is often desk rejected because it's judged not to be significant or interersting enough (usually the author comes from a less prominent university).
For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
This is interesting. Has there been a write up on this saga anywhere?
I'm so glad to be out of that world, to be honest.
This problem goes even deeper, because it not only applies to papers but also to research grant applications. I even had once got a (positive) recommendation for one of my applications, but I could see from the statement that the reviewer did not really understand what was important.
That's not to say the proposal isn't worthwhile, just that it didn't find the right audience. A contractor may have a whiz-bang new method of snow removal, but it doesn't mean people in Phoenix will want it (or even understand it).
.) an unsolvable problem inherent to all peer reviews?
.) a problem that comes from specific practices, and could be solved by doing peer reviews differently?
I'm just a layman - but if reviews are not done by peers (= experts working in the same field)... who else could properly review a paper?
They are actually proposing that peer reviews are credited:
> Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
And yes, I do think reviewers should be credited. Because it's work, for one, that needed to be done so the paper could be published, and because it would probably indeed incentivize reviewers to do the work correctly.
However, it could incentivize them to reject more papers, in fear of accepting a bad paper and be publicly advertised as a reviewer who reviewed and accepted it.
In those domains, everyone knows everyone else. Even if you hid the name, looking at the writing style, the subject, the choice of analysis software, the type of study. etc would largely give away who the authors are. For example I bet you could figure out that Richard Stallman wrote a particular paper, even if I removed his name from it. Or on HN, how many times have you read a comment and gone, "that sounds like..." even without fully noticing who posted it.
I was a PC member for what is now called NeurIPS, and I remember one particular disagreement with a conference co-chair who advocated blinded reviews.
I have since come to believe that the above objections to blinding are mistaken, and that things should generally be moved to double-blind where at all possible.
What happened is, I participated in blinded review processes (for proposals and for conferences) and they turned out to be different. Just that level of de-coupling -- where you aren't 100% sure that the author/proposer is who you suspect -- turns out to matter a great deal psychologically. It changes the framing. And more important, the results seem better.
Every time this is raised, inevitably people will pipe up and say "Oh, you're just not a <specialist in field>, that's why you can't read the paper!"
For example, a new particle discovery paper was being debated in a forum and several people raised the issue that the paper was impenetrable, and was followed with the usual retorts of "you're not a particle physicist!". There was also a response from a particle physicist who works at CERN. He basically said that he himself could not understand the paper, despite working on a similar experiment on a different instrument. His office was down the hall from the office where the paper was written, but each team has their own terminology, jargon, and instrument-specific calibrations. Essentially, they can review their own papers, but nobody else can meaningfully validate it, or even understand it in a useful way!
The problem with how modern science is performed is that it rewards the act of publishing a paper, not the utility, validity, or comprehensibility of the paper.
In fact, writing papers to superficially look more 'serious' is rewarded, so papers that start off quite readable are often edited to be more obtuse and full of unnecessary symbols, terminology, or equations.
I noticed this in Computer Science, where earlier papers in the field from the 60s and 70s were very readable, but then slowly started getting filled with Greek symbols and obscure terminology for no reason. There is no long history of Ancient Greek Computer Science that needs to be preserved for continuity with papers published hundreds of years ago! This is pure intellectual wankery, and has resulted in people like me unable to follow some modern CS papers on topics that I'm already familiar with.
PS: Imagine the quality of "open source libraries" if everyone got accolades for publishing their code to GitHub, NuGet, or NPM, but no reward for anything else. No in-place updates. No direct usage. No pull requests. No feedback other than a one-time rubber stamp. No issue tracker. Nothing. Just... publish your code once and walk away, you get a cookie. Doesn't even matter if it compiles. Doesn't matter if it's pseudo-code you made up yourself. Just get on there, and you get something. But you get nothing for anything else.
Picture that.
That's modern science.
Right now we have papers simply filled with nonsense (https://link.springer.com/article/10.1007/s12517-021-08021-2) and papers on midi-chlorians (https://www.livescience.com/59927-midi-chlorians-paper-accep...). Even without the needlessly over-complicated writing, some papers may really only be able to be meaningfully reviewed by a small number of experts, but that won't be a huge number. Really I'd be happy if we managed to catch the papers any child in grade school could spot as phony.
Maybe we shouldn't be asking if peer-review is a good idea based on it's performance so far until we give it a serious effort.
Yesterday I started looking for a token embedding method, and my search turned up word2vec and GLoVe. After skimming both papers I am leaning toward word2vec solely because it was easier for me to understand the big idea, even though GLoVe may perform better.
Personally I would rather see code than a LaTeX-typeset equation. With typeset equations, the author has plausible deniability that you didn't interpret their notation just right, or oops that was supposed to be a boldface capital X, which implies a different data type. But if you provide code, anyone can run it and independently verify the results.
Maybe the fact that no one can independently verify your equations/pseudocode is a feature, not a bug.
You can't do much.
Open science addresses a lot of the pain points that are mentioned in the article. I think we can all agree that having certain magazines that include what can be considered the best works is a good thing; people for better or worse need these instruments to seamlessly judge the base quality of an article. Regardless of my personal criticisms and potential declining quality of a particular magazine it is obviously very helpful to be able to know "hey, at least I can expect some degree of quality given that it was published in this or another magazine" specially as a student.
But this does not mean we ought to gate-keep research. Not only that, but having your publication out can open up a lot of feedback that can be incorporated and addressed. And this is where I go back to my original statement, in ML, it was very common to publish your work to arxiv for people to read through it and I think that greatly improved the speed upon which the field developed. Access is very important.
So I'm all for abolishing "pre-publication peer review".
Of course we need peer-review and neither me nor the paper in the OP say otherwise. We just can't be gate-keeping research because of the review process; the review process needs to be re-thought and that includes the when and why.
I believe that double-blind, non-public reviews do a great job at protecting those in a more precarious situation: grad students are not judged by their lack of publication record, there is no permanent public record of all those times they got rejected, and reviewers can be both more honest and more certain that a rejection won't lead to a 4Chan/Twitter mob coming for them.
According to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents". I'd be weary of implementing any changed that would make academia even more hostile to women and minorities.
[1] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...
Sure, people like citing themselves. Editors of double-blind journals already are supposed to desk reject work that says something like "in our prior study (Smith 2020)..." They should also desk reject obnoxiously self-referential citations --- like "Great advances have been made in method X (Smith 2016, 2017a, 2017b, 2018, 2019a, 2019b, 2020, 2021)"
But my own impression and experience is that your statement is naive. The key thing I think you miss is that researchers/labs/groups develop fairly specific interests, expertise and "research programs". So taking any paper and looking at the specific area or phenomenon being probed, the techniques/methods used, and the questions or interests driving the research and its analysis, and you'll get a pretty unique "fingerprint" of the researchers that anyone in the same field should be able to identify with pretty good accuracy.
The naivety of your statement I think is this idea that experiments or acts of research are kind of plucked from the air by any who are interested and then the method or technique required is simply downloaded and executed like software.
Researchers aren't generic like computing hardware tries to be. Far from it. A researcher and their research or ability to do research is much more like a tree ... grown organically through experience and pretty fixed and inflexible once it's mature.
None of this is to be really critical of your statement. In fact, I think it's very useful (provided I'm onto something and not merely bitter) to state the ideal practice of science and contrast it with reality.
That we can't do double-blind review is almost certainly a bad thing. And it also tells us something about what kind of "science" (here the global collective enterprise) we have. I think I'm correct about the "fixation" of researchers. But I also think that the culture of science we have encourages this process. I've seen it myself: researchers trying a minor change in field and technique (very minor) and hitting massive resistance from the funding agencies. The conclusion we all drew from seeing it was that the techniques you can get funding for are mostly fixed by your first 10 papers or so. This could be changed and research could get closer to what you describe and I think that would be a good thing. But, as I stated in another comment, to get there requires change beyond the peer review process.
Something else it tells us is that our science operates in a relatively closed and inward looking cultural ecosystem, which indicates to me some degree of corruption is almost guaranteed. If we don't have good diversity and competitive and corrective interactions across space/culture, then, as the "coffins in the ground" adage tells us, we have to rely on diversity across time/generations. And indeed researchers know this just from experience. Someone winning a Nobel prize is like carving their thought in stone ... it takes time for us to move on.
An interesting study in this regard might science during the USSR. I've read glancingly that mathematics research progressed differently between the West and USSR and that, for this interested, cross-pollination was fruitful. An analysis of this could prove insightful. I wonder whether something similar is going on in China at the moment.
At least for mathematics papers, the community who works in a particular area, and is thus actually able to do a review the paper, is pretty small. Since, on the other hand, the style that you use for mathematical proofs is like a fingerprint, double-blind peer reviews are next to impossible.
This made me curious about how arxiv operates. It seems that you require endorsement to become a registered author, and the submissions are moderated[1][2].
This already seems sufficient to keep out spam and clearly junk science. What is the value add of official(?) journals?
(Disclosure: I volunteer for https://plaudit.pub, a non-profit that aims to separate that from the publication process.)
Edit: Here's a proof of the Riemann hypothesis submitted just last week: https://arxiv.org/pdf/2209.01890.pdf You can have a little fun exploring "proofs" of famous theorems on the arXiv over the years.
> Where is that quote from?
by providing a link to the source of the quote.
Ironically, maybe the person to whom you first replied was actually intending to respond to the comment you just linked instead of making a top-level comment.
* Theoretical work * Computer simulations (re-run the simulation? Well that's not going to detect most issues. Re-create the program from scratch? expensive and hard to make a re-write meaningfully different systematically) * Cosmological observations (can't re-run that supernova!) * Medical case studies (can't just find a new patient) * Large scale studies (build a second LHC? Do results based on the Framingham Study have to kick off another 30 years of data collection before they publish)
In fact it's only small and self contained experiments where this is really practical.
It would have detected the fraud going on at ICL with their covid model that was used by the governments of the world to set the course of pandemic response. I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. Running the same simulation with the same inputs produced wildly different results between runs. That would have been an easily detected red flag but because no one was able to do so until long after the pandemic response was set in motion and changes would have been politically uncomfortable for the powers that be, we ended with a terrible response that many still cannot bring themselves to admit was both poor and instigated by a terrible model that should have been easily detected as being garbage.
> Among the two popular models, we found that the ICL model is more transparent and reproducible compared to the IHME model. The former sometimes over-predicted future deaths while the latter clearly under-predicted post-peak deaths. Both models predicted the timing of peaks reasonably well using data until one week prior. The ICL model produced a much wider band of uncertainty for New York state, possibly because the pattern did not conform well with their internal training data used from European countries.
Nature (on 08 June 2020) reported "a computational neuroscientist has reported that he has independently rerun the simulation and reproduced its results. And other scientists have told Nature that they had already privately verified that the code is reproducible." - https://www.nature.com/articles/d41586-020-01685-y citing the CODECHECK certificate at https://zenodo.org/record/3865491 from May 29, 2020.
You should also be asking why public health is so under-funded world-wide that many countries looked to the ICL model - which in turn was a repurposed influenza model - rather than use local expertise or be able to use better validated models.
> Running the same simulation with the same inputs produced wildly different results between runs.
As I recall, the replication issues were the normal issues related to parallelization and random number generation, and not things which greatly affected the results. There were a number of HN threads on the topic back in 2020. Eg, https://news.ycombinator.com/item?id=23212268 , which linked to a couple of those issues and their resolution.
> I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here.
I can. The Reinhart-Rogoff spreadsheet error which was widely used as the excuse for "austerity" cuts, rather than the real reason of wanting to lower taxes on the rich. https://truthout.org/articles/excel-spreadsheet-error-lesson...
I wonder where that tax money could have been used. Perhaps into public health measures?
Which is why reproducibility should be the most important topic.
You will be amazed what you didn't spot/assumed and learn a lot.
There also often isn't a clear cut line between "good" and "bad". Almost all papers have flaws and putting those out in the open for others to improve upon, or at least acknowledge, would help move humanities knowledge forward.
it would enable faster and more seamless communication, academics wouldn't be burdened with extra volunteer work, and publishers would still be able to curate works as the authors suggest (+ there are other ways to do it).
sadly my impression is that it's simply too entrenched in the publishing process -- and publisher's raison d'etre to a large extent -- for the publishers to relinquish control of it
I read an interview with a TV show runner who said you can come up with any crazy plot twist thing, and within 10 minutes of the first episode airing some guy on the internet has already figured the whole thing out. I think this phenomenon could be put to good use.
Just publish your papers publicly with a comments section. If there's problems with it people will tear it apart. Source their work and let the world help you improve your work.