Remember that Einstein told journal editors to piss off [1] when they tried to get his papers peer reviewed.
[1] https://theconversation.com/hate-the-peer-review-process-ein...
What the people who critique the publication process are missing: 90% of submissions are crap - unfit for publication.
We need some process to gate-keep.
a) The venue of publication is a good signal, whether the time to read a paper is well-spent. b) The PhD students learning the craft need objective feedback. The supervising professior/university often has the incentive to "just submit" -- even if they know that a publication does not meet the quality standard.
Before peer-review, somebody also needed to make a decision on what to publish. This typically fell to a single individual. The editor or some well-known member of the community who could recommend a paper for publication. On old journal issues they even mention the "recommender".
So the question is not whether peer-review is bad, the question is which alternative gate-keeping process would be better. Otherwise we will drown in crap publications (even more) and the PhD students don't get a honest feedback signal upon which they can improve their craft.
> We need some process to gate-keep.
Journals, when print was the medium through which academic research was disseminated, had to gatekeep because there were practical considerations regarding how many articles they could put in each issue. With online repositories like arxiv, this is hardly a concern anymore.
Someone putting a crap article on arxiv does not hurt anyone else, and I'm saying this as a person who recommended tons of articles to be rejected because they had atrocious grammar/spelling issues. Worst case, it gets 0 attention and is ignored by the research community.
Something not being published in a journal/conference proceedings clearly does not prevent it from drawing tons of research attention, as we saw in numerous cases like the Adam optimizer [1].
Which brings us to the second point: what even is the purpose of a journal now? The answer is that the sole function of a journal now is gatekeeping, with the presupposition that, as you observed
> The venue of publication is a good signal, whether the time to read a paper is well-spent
Except, well, top journals have tons of articles that get 0 citations too. Clearly the filter fails at this purpose as well. So, why gatekeep at all then? Because if we did not have some exclusive prestigious journal, the plebs would not be separated from the esteemed titans of academia with the biggest grants, most prestigious scholarships and diplomas from the most famous universities.
The only reason we need to gatekeep today is to feed the academic prestige and politics machine. If you care about the science, upload the goddamn PDF to arxiv , tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.
That said, there definitely are very relevant papers that are not published in any peer-reviewed venue. A good example is "Language Models are Unsupervised Multitask Learners" (the GPT-2 paper, which I would argue started the whole generative LLM revolution). But I think if you look for this kind of papers, you will find something in common to all of them: they are by very well known researchers, elite institutions or influential companies. That's why people went out of their way to read them even if they were posted somewhere without peer review.
If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact. It's peer review that allows them to be able to submit to a top journal, where the editor and reviewers will read their paper, and they can get a somewhat fair chance.
PS: I don't really like the peer review system that much either. It's just that every alternative that I have seen proposed so far is worse.
Fair, must have misremembered that one.
> If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact.
I disagree on this one. I did my PhD at an institution that ranks in the top 10 in the most well known university rankings, and I distinctly remember that one time when I was submitting a manuscript to a prominent journal in my field, got some reviews back which weren't positive yet were quite valid criticisms, and my professor told me not to worry because the editor is his buddy and my manuscript will get published for sure.
When that sort of "scratch my back and I'll scratch yours" culture exists in journals I don't see how peer review can be an equalizer. It just means everyone who publishes at a journal, including the less well esteemed ones, can claim they went through the rigor of peer review. Of course, we all know peer review is just a vibe check and is actually not that rigorous at all, and besides no one cares unless you published in a prestigious journal anyway. The less revered journals exist to collect $5k in open access fees for the publisher in return for hosting a pdf at the marginal cost of maybe a cent a year.
[0] https://elifesciences.org/inside-elife/66d43597/elife-s-new-...
So the role of journals and conferences is not to prevent "the word getting out". It is to provide value by a curated list of on-topic and high-quality publications.
So no need to wade through tons of crap. Especially for PhD students who might take more time to detect crap as such.
In my experience, the publications at the good venues get a lot more eye-balls and by consequence citations. So there seem to be a lot of people who like this role as a "filter" for what to focus on.
> ... only reason we need to gatekeep today is to feed the academic prestige and politics machine
This to me says you have may not have experienced some parts of the (long-term) research process. It suggests that you have infinite resources to filter out noise, which is probably not the case. It suggests you're willing to spend a lot more time figuring out why something doesn't quite make sense, rather than get to the heart of the problem, while this is fine in many cases it sucks when you're hot on the trail of something interesting, and you're slammed by a million twisty paths full half-baked hot-takes.
We need to filter ("gate-keep" is pretty inflammatory term) information and processes so that we don't have 12 different screw types with 12 different electric screwdrivers, instead of "just" 6 (sigh). We need to come to consensus and that means some things go in and something are left out. We need many mechanisms to filter.
> tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.
All of these things feel like filters, when does a filter become a gate: colleagues - i.e. not everybody, but some selected few, who and how?; conference - filter (well, gate!); scientific community - != your baker's community; decisions directed by you, not on my own (i.e. a pointer to my paper) - filter.
[Edit for formatting, sort of.]
As for the "scientific community" being a filter, there is a difference between "elite" researchers being the ultimate arbiters of scientific truth via their positions in the editorial teams of journals versus everyone being allowed to publish on open platforms like arxiv and bad ideas/quackery being filtered out naturally.
Because the former is what makes or breaks a scientist's career, grad students and postdocs hyper optimize to publish at prestigious venues, as opposed to optimizing for doing science. These two are aligned only sometimes.
Per Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." [1]
[1] https://archive.org/details/ImprovingRatingsAuditInTheBritis...
Nor do journals.
I think it's straightforward to make an argument that many of today's conferences are as bad as journals, accepting submissions is only one way conferences filter. IRL they are prohibitively expensive to enter, let alone attend (but again, see "Zoom"), and therefor eliminate all but the elite, they are run by commercial entities in all cases with more than ~200 people, they are more or less required venues for networking and therefor selling yourself for a tiny chance at academic permanence, they give plenaries to elites (filtering to one voice), they have special symposia by invite only, with other submissions dumped to inaccessible parallel sessions (which one will you choose to see?), the submissions you make are published in much more ephemeral ways, and tend to be more difficult to discover in the long term, making the event important but the research not as much (at least in my experience), etc.
But having a browsable collection of the verified non-crap articles on any given topic helps most everyone working in that area.
Publication is antiquated. HN doesn't need reviewers to boost the best content or to provide commentary on how to improve a paper or fact check its contents. Join the 21st century.
Sorry, don’t mistake my tone for condescension. I just wanted to explain how this looks to an outside observer.
Academic politics always existed of course. We should not be under any illusions that the greats of the past just wrote their manuscripts in isolation. However it was not an industrialized machine like it is now, and incentives are misaligned in academia to such a degree that the machination of academic politics killed the reason why academia exists in the first place (scientific discovery).
To advance in the academic cursus honorum, one goes to a presitgious undergrad to go to a prestigious grad school, so you can get a prestigious postdoc grant, so you can get a tenured position at a prestigious institution, so you can get a fat and prestigious government grant, which you use to hire bright young students who want to do the same. Note that scientific advancement does not play a role in this cycle, it's actually safer to pursue incremental and irrelevant improvements which you get published through thanks to the connections you made throughout your prestige optimization career.
As a result, academia has produced no notable scientific advances in a long time. It has instead evolved into an organism which selects individuals that excel at funneling money into itself under the guise of doing science while not necessarily doing science (though it incidentally occasionally may).
The kind of behaviour the author is promoting is telling individual prospective grad students you're small, the academic politics machine is so big, yes we all know it's a farce but you just need to suck it up and play the game. In doing so, the prospective grad students will strengthen the machine that is actually killing the very thing they want to cherish and promote, in the hope that they receive a few scraps in return.
(Writing all this as a PhD and former journal reviewer)
He was just as wrong then, as you are now.
1. Finish
2. Finish
3. Finish