Comment by D-Machine | Hacker News Reader

Comment by D-Machine | Hacker News Reader

The other factor preventing a fix is that people with no actual serious experience of academic publishing and peer review will defend these journals, because they still think that (journal-based) peer review acts like some kind of meaningful quality filter. But, it really doesn't.

Because someone is surely going to try to defend journals via peer review in this thread, I want to provide a counter to the arguments that journal peer review does much good. Also, since everyone knows that if you just go to a poor enough journal, you can be published, I am going to focus on the (IMO mostly false) claim that higher-profile journals are still doing a good thing here.

There are numerous studies showing that higher-profile journals in general have more retractions and research misconduct [1-2], lower research quality [3], in fact weaker statistical power and reliability [4], and that statistical reliability even in high prestige journals is still extremely poor overall [5]. Also, making it through peer review is highly random and dependent on who you get as a reviewer [6], or is just basically a coin toss even when looking at reviewer groups:

    In 2014, 49.5% of the papers accepted by the first committee were rejected by the second (with a fairly wide confidence interval as the experiment included only 116 papers).  This year, this number was 50.6%.  We can also look at the probability that a randomly chosen rejected paper would have been accepted if it were re-reviewed.  This number was 14.9% this year, compared to 17.5% in 2014. [7]

We should just move to arXiv-like approaches and allow the scientific community to broadly judge relevance and quality. Journals just slow things down and burn funding for very little gain or benefit to anyone other than the journal owners.

[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC3187237/

[2] https://www.pnas.org/doi/10.1073/pnas.1212247109

[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC9382220/

[4] https://journals.plos.org/plosbiology/article?id=10.1371%2Fj...

[5] https://www.frontiersin.org/journals/human-neuroscience/arti...

[6] https://journals.plos.org/plosone/article?id=10.1371%2Fjourn...

[7] https://blog.neurips.cc/2021/12/08/the-neurips-2021-consiste...

> higher-profile journals in general have more retractions and research misconduct [1-2]

Given that reviews are not a mechanism to check for truth but soundness, the higher profile the thing I would imagine there would be more misconduct. I mean would one risk prison to steal 10$ or to steal 1 million $?

> lower research quality [3]

To cite exactly from your link "the evidence is mixed about whether they are strongly correlated with indicators of research quality.". I find saying "lower" a bit too strong given the original quote.

> in fact weaker statistical power and reliability [4]

For a specific field "cognitive neuroscience and psychology papers published recently"!

> statistical reliability even in high prestige journals is still extremely poor overall [5]

According to https://www.frontiersin.org/journals/human-neuroscience/arti... they kind of targeted bio/medical/psychology field for the analysis. Which seems to me very focused to be able to draw general conclusions.

> Also, making it through peer review is highly random and dependent on who you get as a reviewer [6], or is just basically a coin toss even when looking at reviewer groups:

It's a coin toss if paper could get accepted at all, and that's less than ideal but what the system should do (at least) is reject obvious crap, not ensure that something gets clearly accepted. The danger is False Positive (accepted even if it's crap) rather than False Negative (rejected even if it might be something useful).

Overall note: the review system is not ideal and should be improved. But it's a hard, complex and delicate problem.

Oh, I agree this is all super complex and delicate. If I had more time, I'd love to write a more nuanced, many-thousands-of-words blog post going into which journals and fields actually have good peer review and can be more / less trusted.

I just wanted to make a strong rhetorical case by highlighting some things that might be surprising to people making more naive defenses of journals via peer-review-based arguments.

I am sympathetic to the argument you wish to make, that peer review is no panacea, but the actual evidence you offer has nothing to do with this claim.

You are trying to say that high profile journals have more retractions, which is well known as you share.

How does that have anything to do with peer review? Are you saying that there is more review or less review in some cases and that influences retraction rate? In what evidence? In what world does the arxiv system moderate this discrepancy?

> How does that have anything to do with peer review?

I already addressed this. People know peer review can be bad, but some think "good journals" still do good peer review. This is not so clear.

> In what world does the arxiv system moderate this discrepancy?

Open systems allow the scientific community to figure out ways to properly assess research quality and value more cheaply, and without passing through (often arbitrary and random) small numbers of gatekeepers that don't even do a reliable or good job gatekeeping in the first place.

Your argument depends on worse peer review at top journals - but fundamentally, you fail to show how doing any peer review is strictly worse than doing no peer review.

I understand that we want arxiv to exist, and it does, and it’s growing. That doesn’t mean we don’t want Nature or Science to triage the most compelling stories.

Importantly, we can already begin the search for these ‘cheaper’ review strategies while not losing the helpful information filter we get by seeing where things are presented/published

> Your argument depends on worse peer review at top journals - but fundamentally, you fail to show how doing any peer review is strictly worse than doing no peer review.

No, it doesn't. The argument is that peer review is incompetent gatekeeping in general, and so slows things down and makes thing expensive. Also, I am countering the argument "we need journals because journals do peer review" by arguing "peer review by journals isn't clearly actually good", I am not saying "peer review in general is unneeded", as I support review by the entire scientific community, rather than journal gatekeepers.

> you fail to show how doing any peer review is strictly worse than doing no peer review

I wasn't trying to show that. I have provided plenty of arguments to show why killing journal-based peer review could definitely speed things up and so potentially make things better. I want actual organic review by the community, not by tiny groups of gatekeepers.

> I want actual organic review by the community, not by tiny groups of gatekeepers.

But this happens—and good work is cited and talked about. I can't tell if you work in science, but this latter part is obvious.

I do work in science, I am claiming that pre-publication / journalistic peer review is limiting (and biasing) the amount of post-publication / non-journalistic peer review that can happen, and it is not limiting this in a very reliable or even IMO particularly desirable way.

There is definitely a problem with the over-production of junk science, and we definitely need a way to filter this out somehow. I am just claiming journalistic / pre-publication peer review does not do this effectively or reliably at all anymore (if it ever did).

Reply on news.ycombinator.com