Hacker News Reader

Hi! Author here.

Preprint review as it is being discussed here is post-publication. The preprint is shared first, and review layered on top of it later. Click through to the paper[1] I'm responding to and give it a read.

But, also, prepublication review doesn't need to be "reintroduced". It's still the standard for the vast majority of scholarship. By some estimates there are around 5 million scholarly papers published per year. There are only about 10 - 20 million preprints published total over the past 30 years since Arxiv's introduction.

There are a bunch of layered institutions and structures that are maintaining it as the standard. I don't have data for it to hand, but my understanding is that the vast majority of preprints go on to be published in a journal with pre-publication review. And as far as most of the institutions are concerned, papers aren't considered valid until they have published in a journal with prepublication review.

There is a significant movement pushing for the adoption of preprint review as an alternative to journal publishing with the hope that it can begin a change to this situation.

The idea is that preprint review offers a similar level of quality control as journal review (which, most reformers would agree is not much) and could theoretically replace it in those institutional structures. That would, at least invert the current process: with papers being shared immediately and review coming later after the results were shared openly.

[1] https://journals.plos.org/plosbiology/article?id=10.1371/jou...

Thanks for your charitable response to a reader who quite obviously misunderstood what you were writing about. You might want to update your article to make that clear, as "preprint review" isn't a term I had ever encountered before, and a simple reading of your article doesn't obviously indicate that you were talking about post-publication review. (Note that outside of academia, upload to arXiv is publication, so yes your "preprint review" would be a post-publication review.)

No worries. I was really writing for an audience of academics and, in particular, people involved in the various open science and publication reform movements. Sharing here was a bit of an after thought, so it wasn't written for an audience unfamiliar with the intricacies of academic publishing. It's a complicated enough space as it is that this is already a 4-5 article series with the potential to grow, even when I'm writing with the assumption of quite of a bit of knowledge.

But yeah, "preprint review" is considered post-publication review both in and outside of academia. There are nuances to what is consider "publication" in academia. A preprint is not a "Version of Record", meaning it doesn't count towards tenure and promotion. The movement pushing for preprint review is attempting to layer review on top of already public preprints in the hopes that reviewed preprints can begin to count as VORs. It's unclear whether that will work.

Some models, like eLife's, seem more promising than others. But eLife got a ton of backlash when they switched to their new reviewed preprint model, so it remains to be seen whether it will work in the long run.

Quality control was handled fine enough by editors for literally all output on the planet pre 1970s.

There’s nothing physically stopping that paradigm from returning now that we have the internet, other than the fact that there’s only a few thousand (?) such editors.

If somehow there were fifty thousand such editors, then the whole peer review system would be completely unnecessary.

Of course not enough people want to pay for that many editors, but that doesn’t stop a partial adoption by those willing to do so via some arrangements.

There are nearly 50,000 commercial journals and a long tail of non-commercial journals each with teams of editors. There are probably hundreds of thousands of people currently serving as editors.

The issue isn't editorial bandwidth, it's that peer review is currently built into the promotion and tenure structure for academics, who produce the vast majority of scholarship and thus dictate the shape of the scholarly publishing system.

Academics have to publish the papers in peer reviewed journals for it to count towards tenure and promotion. And in fact, they are limited to a small set of journals that are deemed high quality enough for their fields. These journals are chosen by tenure and promotion committees composed of their senior peers and school administration. There are over 1000 R1 and R2 universities worldwide, each with hundreds of departments each with their own tenure and promotion committees. So changing the system is a massive collective action problem.

I’m including only actually competent, full time editors, with sufficiently high reputation that their decisions will be taken seriously. There’s definitely not 50000 of those.

A huge number of journals by numerical count, along with their ‘editors’, are literally laughed at in many fields.

As you’ve mentioned, trying to expand the actually reputable number by 10x, 20x, etc… is a huge problem.

Hence it has to be paid for, quite highly paid for, otherwise the coordination problem is probably impossibly difficult.

You are right. It is often a problem that established competitors get to review articles that contradict their work and, unsurprisingly, try to sabotage them. Incentives are not well aligned.

A good mid-ground is something like the non-profit journal ELife, where articles are accepted by an editor and published before review, then reviewed publicly by selected reviewers.

Very transparant, and also leaves room for article updates. See the whole process here: https://elifesciences.org/about/peer-review.

That’s a better system. But why involve the journal in review at all?

Journals should go back to just publishing papers and any unsolicited letter-to-the-editor reviews, reproductions, or commentary they receive in response. Why add a burden of unpaid work reviewing every single paper that comes through?

I believe ELife may eventually get deeper paid reviews. That is a reason to involve the journal in this process. The way reviews at ELife work can be seen as solicited letter-to-the editor reviews, as these do not influence the outcome. Your article is already published.

Maybe I'm not explaining my point very well. I think the fact that some papers don't get reviewed, even post-publication, is a feature not a bug. Otherwise it starts to present an unnecessary and damaging hurdle to publication, even if the reviews are post-publication because it is an implicit promise of future work.

A proper scientific journal is an efficient clearinghouse for information, in the same way that Hacker News is an efficient clearinghouse for tech news and commentary. Journal editors play the role of dang in this setup, applying a minimal but necessary amount of moderation and editorial decisions. But imagine if you weren't allowed to post a comment to HN unless you first lined up 2-3 high-karma individuals to provide a thoughtful reply. No doubt the result would be high quality discussion, but there'd only be a handful of comments on even front-page posts, and I suspect that the overall value of the site would be vastly less than the present HN.

There should be as few as possible editorial hurdles to clear in order to publish in a journal, e.g. one of the co-authors having published in the journal before or being sponsored by a respected person in the field. And the review that occurs before publication should consist of (1) spellchecks and such, and (2) formatting. This is how science worked for hundreds of years, and there's no reason we can't continue operating this way now that journal page lengths are a non-issue.

(I'm not a reactionary though. I'd like to improve upon the old format in many ways. In particular I'd want journals to provide specific support for publishing reproductions of existing work or 3rd party submitted supporting documents, and much better methods for retracting or correcting a paper.)

I've found the entire peer review concept in academia to be extremely odd. I'm not entirely sure what problem it solves. It seems like you have one of two types of people reading these articles:

* People who are already specialists/familiar enough with concepts. They'll either call BS upfront or run experimentation themselves to validate results.

* People who aren't specialists and will need to corroborate evidence against other sources.

My entire life as a software engineer has been built blogs, forums, and discussion from "random" people doing exactly the above.

Most of the world isn’t ‘about’ software engineering. Software has some attributes that not very many other fields do. And software people have a habit of thinking that their experiencing writing software grants them some sort of transferable expertise, which it certainly does not.

> run experimentation themselves to validate results

Let's just breeze through an that like it's nothing haha.

Also the people reading these peer reviewed articles range from new grad students to researchers with decades of experience.

There are many ways to see whether an article is high quality of not, which includes peer review, the journal it's published in, the research lab that wrote the paper. Reading a paper itself is a multi hour ordeal and you want to have a decent idea that it's not a crap paper before diving in. Believing what the paper says is something of a gamble because you really cannot just replicate an experiment that easily. And you need to read many many many papers before you can start doing your own research. So you want lots of assurances that a paper is good quality.

Agree. With more compute-intensive experiments in ML, I am not sure if any reviewer re-runs experiments. The reviewing is already voluntary - who's going to pay out of their pocket for the compute? It's not feasible.

Re-running experiments is out of scope for any reviewer. You're supposed to trust their data, but review that their conclusion is actually supported by the data, as shown.

Yeah, I agree. But I said what I did to make the larger point that empirical studies are hard to vouch for without some level of trust on both sides. For ex if someone selects favorable datasets, metrics, "random" seeds, p-hacking etc.

https://en.wikipedia.org/wiki/Replication_crisis

The reputation of scientific researchers has been greatly harmed by the current system. Please, help find a way to fix it, or at the very least don't hinder people trying to fix it. Thanks to the way we do things now a coin flip is _better_ than peer review. Public trust in science is at an all time low. I really hope you don't think "this is fine".

IMO 'public trust in science is at an all time low' is because of bad journalism more than bad papers.

Papers are written by academy-type individuals for academy-type individuals, not for consumption by non experts. An academic is usually pretty fast to determine if a paper is to be trusted.

So interpreting and extrapolating to the extreme the results of a minor paper in an obscure journal is more bad journalism than bad science.

Then we wonder why people don't trust science..

> Thanks to the way we do things now a coin flip is _better_ than peer review. Public trust in science is at an all time low. I really hope you don't think "this is fine".

I don't think you read my post? I'm advocating we get rid of the "peer review" [sic] system entirely.

The sibling post is right though that this problem is with bad journalism (and bad institutions), not bad science. People think that "peer review" is actually some sort of scientific hurdle that strengthens the paper. It is not, it was never meant tho fill that role, and has been totally morphed by journalists into something it has no business being.

Also, the author IMHO failed to clearly explain how "preprint review" wasn't a contradiction in terms (though they do seem to gesture towards the main issue being commercialization of journals, in the first post).

In the same vein, the positive mention of Github and "open source platform" (another contradiction in terms) were at first red flags in the third article, but at least they then mentioned the threat of corporate takeover.

> The idea of having your intellectual competitors review your work potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective.

If this is true (and I'm not doubting you, just acknowledging that I'm taking your word for it) then why abandon the entire system? Why not just roll it back to the state before we added the intellectual competitor review?

>> Why not just roll it back to the state before we added the intellectual competitor review?

Journals don't add much value outside of their peer review.

Most researcher don't care about the paper copies, or pagination, or document formatting services provided by publishers. Their old paper based distribution channels are simply not used.

The prior state was a situation of no pre-publication review other than the editorial staff of the journal. We should go back to that, yes. By disbanding entirely the “peer review” system that currently exists.

>Why not just roll it back to the state before we added the intellectual competitor review?

The only thing I can currently think of is that the pace of research* has grown so much that a small group of editors may be unable to handle the amount of submissions. This could result in a) an inability of the editors to thoroughly vet the submissions, b) difficulty in "good" submissions being found (ie, separating the wheat from the chafe), or c) a further devolution into very, very niche journals just to make the scope manageable for the editors

* I would concede that a very, very large proportion of current research is either heavily derivative or auto-cited, so overall growth isn't to be conflated with growth in quality research

You are mistaken in thinking that editors are supposed to vet submissions. That's not their job! Their job is only to weed out crackpot pseudoscience, which is a much easier task generally and also mostly solved by reputation, or papers outside of the journal's scope. And to edit the remaining papers into a constantly formatted journal publication. And that's done easy enough these days with LaTeX and online printing.

It is NOT the job of journal editors to rate or vet sincere submissions they receive that are on topic for their journal. This only started happening about half a century ago, when demand for publication started to significantly outstrip supply of journal pages, back when journals were actually printed on dead trees and had limited number of pages to keep costs down. The idea then was "we're getting 50 submissions but can only print 12, so let's rate them and pick the best ones." So they started the 'peer review' [sic] process to externalize that vetting cost. It largely didn't exist before then. Only now we can accept all 50, because why not? The marginal cost of one more PDF is practically nil.

>Only now we can accept all 50, because why not?

Because the downside to creating an ever growing haystack is that it becomes increasingly difficult to find a needle. Making it easier to create a deluge of bad research won’t help me find the worthwhile research that would actually help me in my job.

If I had the choice between collecting “all the data” and just collecting “the really good and relevant data” I’m opting for the latter. You are also contradicting yourself by saying it’s not the editors job to vet submissions, yet also say they weed out “crackpot” work. All you’re saying is they lazily/loosely vet submissions. I’m saying the overall system (not just the editors) have a role in providing a reasonably sized haystack (and would also admit the current system is not great at this, but it’s better than a wild-west approach)

You just use better tools to manage it. Fine-tuned LLMs and Google Scholar like search engines help here.

To stretch an analogy it is like email. The job of the editor is the same as the spam detection service run by hosted email providers. They actually go in and actively hide scams and worthless ad email from you, and we thank them for it. Some email providers have recently started offering "focused inbox" modes where they prioritize emails for you too. I don't use that, but I could see why some people do. But importantly they don't block email based on those heuristics, like they might do for spam. You still get non-priority emails. But imagine a world where gmail straight up blocked/rejected email which it didn't consider priority. Would you want that?

The situation with journals is comparable. Editors have a spam/crank detection duty, but they shouldn't be rejecting manuscripts beyond that.

What you’re describing is essentially an arms race in quantity. Yes, we can use tools to help sort, but those same tools can also be used to deluge the inbox and obfuscate the bad. In fact, one of the best ways to sort is by using specific journals/journal metrics as a proxy for quality. That is much, much easier (and productive) than trying to sort based on some Google scholar advanced query. For example, it's much easier for a journal to retract an article that was shown an inability to replicate than to create a search to do the same.

The tone of your comment is very techno-optimist, which is very on brand for HN. In that view, every problem is solved by technology, even those that are created by technology. I would argue there are some problems that are better solved with less technology, not more.

> Editors have a spam/crank detection duty, but they shouldn't be rejecting manuscripts beyond that.

If the system is working, publication in a reputable journal serves as a useful, albeit imperfect, indicator of scientific quality.

Top journals shouldn't be publishing deeply flawed work, or even decent work in clear need of a rewrite. It's not just about spam and cranks.

The whole process comes from a time when publishing was expensive, should be cheap as chips now. The system needs a rethink so "quality" can somehow bubble up to the surface given mass publication is so simple.