Hacker News Reader

Your points are mostly fair, though I'll push back on one. For context, I'm a late-year PhD student in ML, but I primarily do theory.

> Reviewers might not have relevant expertise to judge whether a paper is good or not...

Finding enough quality reviewers is hard. There are so many papers (heck, you put out 30 papers in 3 years!) and there is not much incentive to review. I review maybe 5x more papers than I write each year. It's doubly hard to get good reviews out of senior people. For this reason, I think grad student reviews are not a bad solution. In my experience, grad students puts in more effort and write decent, careful reviews. The very worst, low-effort reviews seem to come from established people who can't be bothered. A lower reviewer bar at workshops is fairly common and reasonable. Workshops are not the main publication venue (at least in my areas), so they mostly just want to filter out papers that are bad rather than only accept papers that are good.

> Politics...

I can't say too much about this since I primarily work in theory, but my perception from the outside is that the more "lab"-dominated a field is (i.e., accepted papers tend to come from some group with 20 PhD students and 10 postdocs) the more likely there are to be "mafias". Maybe this is a function of the kind of people who want to head up a lab, maybe it's a function of the kind of work that gets produced in factory environments, or maybe it's baseless prejudice from me.

You are probably right about postgrads. Although i would not consider myself the best reviewer i was really investing time and effort to produce a fair and honest review with the "limited" knowledge of the field i had, and i was probably a better reviewer than "experts" who would produce 1 line reviews or generic comments "add more references". However i would beg for highly-technical reviews, providing harsh theoretical and technical feedback, which i guess would come from more experienced or expert reviewers than postgrads or myself. And by expert i mean people who have knowledge and publications on that specific paper topic, e.g. you might not be able to provide expert feedback on P2P or distributed networks just because you are an expert in QoS (network buzzwords here, please ignore :) ). Which is probably an effect of what you mentioned; the amount of conferences/workshops/journals and papers submitted and lack of incentive from people to review.

The ~30 papers were an outcome of collaborative R&D projects with universities and companies, most of those just having my name in the authors and requiring me to quickly read and comment them, instead of spending effort to produce the results presented. From a third person perspective this somehow justifies the 3rd point of my original post on how these mafias are created and maintained. The thing is that i didn't care for citations or more paper invitations because my position or wage didn't depend on them (i was just doing my research for the implementation and the trips in the conferences my papers were accepted) and eventually i am not doing R&D anymore, however i can imagine how unfair this whole setup is for people who want to follow an academic path.

> However i would beg for highly-technical reviews, providing harsh theoretical and technical feedback, which i guess would come from more experienced or expert reviewers than postgrads or myself.

Grad students are plenty capable of writing these reviews. They might even be responsible for most of them. But the problem is still that there's very little incentive to write these kinds of reviews. As a reviewer, I don't even know how the author will accept it -- will they engage with the criticism and re-work their paper? Or will they ignore it, mildly re-write the introduction, and then re-submit to the next conference? I have seen enough examples of the latter to blunt my enthusiasm as a reviewer for providing detailed constructive feedback. Unfortunately, I think the most useful feedback may come after acceptance, when you give the talk/present the poster and people actually pay attention. It's much easier to interact in good faith when the person is right in front of you.

I sympathize with relative "outsiders" to a field who can only get expert attention through reviews. The above issue makes things difficult. Attending workshops can be useful for this.

> The ~30 papers were an outcome of collaborative R&D projects with universities and companies, most of those just having my name in the authors and requiring me to quickly read and comment them, instead of spending effort to produce the results presented.

Ah, this is not the norm in theory. Our author order is alphabetical, so the working assumption is that every co-author has seriously contributed to at least one (and hopefully all) of 1) original idea for the paper 2) technical work (proving a result, implementing an algorithm) 3) actually writing the paper. There are still people who publish 20 papers a year, but this is rare.

Yeah I’m in cryptography and our papers typically have 1-3 authors. Meanwhile the broader computer security conferences routinely have papers with 8+ authors and I’m thinking “How can 8 people write a paper and not produce total garbage?” Well it’s because most of them didn’t actually write anything in the paper, they just contributed to a code base or data collection.

While peer review has its problems, I have the feeling that the problems that we're talking about here are not primarily caused by peer review per se, but by a set of perverse incentives that surround academia these days. I'm talking about the "publish or perish" culture that is results from the (sometimes) automated metrics based on which promotions are decided, grants are handed out and academic staff is assessed. Isn't it these incentives that can create attitudes like "Who cares? It's one more publication!" (as quoted by another commenter in this thread). And isn't an overwhelming amount of mediocre journal submissions that result from this one of the reasons peer review is under pressure, qualified reviewers are rare and politics are so common?

From the original piece: I don’t have a solution, but I hope my observations here might be interesting to some.

I have a partial solution: researchers "publish" papers to arXiv or similar, then "submit" them to the journal, which conducts peer review. The "journal" is a list of links to papers that it has accepted or verified.

That way, the paper is available to those who find it useful. If a researcher really thinks the peer reviewers are wrong, he/she can state why, and why they're leaving it up, despite the critiques. Peer-review reports can be kept anonymous but can also be appended to the paper, so that readers can decide for themselves whether the peer reviewers' comments are useful or accurate.

If the paper is on the arXiv then it will not be anonymous to the reviewers.

This is how many CS conferences already work (even with public anonymous peer review, go to openreview.net to see it in action). It does seem to be better, although the frustration level for researchers is not really less.

Edit: also, in CS there is a norm that reviewers will not go out of their way to figure out if a submitted paper has been previously posted on arXiv. I don't think it's possible to have perfect anonymity without forbidding researchers from publicly talking about work in progress, so this seems like a reasonable compromise.

I'm in philosophy, though personally working in formal philosophy, so this is quite different from CS. In my area practically all top journals reject more than 95% of all submissions, and if you get accepted, it takes about a year until the article is published. Many journals get so many submissions that they do not provide any reviews or feedback - not even a paragraph or line - if they reject a paper.

Although I have been able to stay a fully paid postdoc for more than 10 years without problems and get decent amount of publications out, I'd agree that the peer review system is broken and seems to be strongly biased in favor of mediocrity over originality. It's particularly bad in fields like philosophy in the humanities, where you could argue that there are no clear-cut criteria for evaluating the content other than merely formal issues. (Philosophy is not a science.) There are lots of people who game the system, I'd say many of my colleagues are doing it, though not (yet) the majority. By gaming the system I mean writing articles they barely endorse themselves, tailored to get past reviews. I've literally had a very successful colleague say "Who cares? It's one more publication!" when I raised doubts about a joint paper, because it involved too much hand-waving and didn't seem to work.

Long story short, I have the impression that in the humanities whether you're accepted or not mainly depends on the writing style and on whether you win the "reviewer lottery." One and the same paper can get accepted in one journal and rejected in another, for journals with similar reputation. I also believe that there are certain niche areas in the humanities that are (almost) pseudo-sciences and based on a small group of researchers colluding with each other and passing opportunities for publication to each other. I won't mention the disciplines, though.

I'm a bit dismayed that you describe CS in a similar way, I've always thought its much more rigid there and easier to evaluate CS papers.

Having said all that, I don't think there is any alternative to the peer review system. Any alternative I've seen so far or come up with myself seems to be even more prone to manipulation and gaming the system. I believe that it's not the peer review system itself, but over-reliance on objective indicators by university administrators that has led to most of the problems. But the humanities have certainly changed to the negative, at least in philosophy I'm certain that many seminal and famous articles from the 70s of last century and earlier would not make it past the peer review today.

sadly, CS is also about the reviewer lottery. however because it is conferences rather than journals, turnaround is much faster at least. and there is a culture of preprints, so having a paper rejected once doesn’t delay getting it out into the world.