Strife at eLife: inside a journal’s quest to upend science publishing
nature.com
nature.com
eLife is the only high profile journal that experiments with the process. Their experiments may be in vain, because they still need to come up with an alternative measure that funding agencies will use to distribute grants. You can't replace peer gatekeeping with nothing, there has to be something else. That said, the "publish then review" model is great, it is a straightforward mechanism that has worked on the internet for decades. We have AI that can weed out the obvious garbage , and the open review process can enhance papers greatly.
I can understand the fears about submission quality. People do prefer submitting to ‘elite’ journals with low acceptance rates, because getting accepted becomes a validation. I wonder if publishing the reviews might take care of that anyway, if the audience will tend to read only the papers with high reviews.
Maybe they could have the best of both by publishing a curated set of the top 10% of papers by review scores in a best-of version of the journal, and still publish everything in the main journal. The move to consider publishing everything makes sense in this new world where all papers are on Arxiv, and they frequently make rounds in the press and social media before getting reviewed. I would welcome the eLife changes if it helps quell the trend of completely unreviewed papers vying for attention.
Reviews are inherently noisy because there are usually no more than three (over-taxed, unpaid) reviewers. But papers are now often highly multidisciplinary. Luck of the draw applies with force at n = 3 or even n = 10.
Reviews DO improve quality of final product, especially in the better journals. But there are still huge gaps in reviewing due to the underlying assumptions about “good” science.
For example, the majority if experimental studies using mice and rats use only a single genotype or strain of animal and are therefore essentially n = 1. My inclination is to reject a great majority of such studies without serious review because they are just fancy case reports.
So if you lucked out and did NOT get me as a reviewer you may be good to go at Science, Nature, or Cell. If you did not luck out then you will have to fight with the editor-in-chief to overcome my fundamental dismissal of case reports in experimental biology.
I support the eLife direction. We have squandered 20 years already and the hegemony gets stronger. Eisen is a bold publication experimentalist.
I'm curious if a large part of this push-back is "I don't want to be held publicly responsible for my review comments..." (the over-taxed, unpaid ones)
... which leads inexorably back to the place publishers don't want to go -- reviewers deserve to be well-compensated, for doing a time-intensive, hard job well. And especially because they create the bulk of a curated journal's value.
I think one interesting model is the one by PLoS (especially with One). As long as the methods are valid, the research is original, and the language is minimally appropriate, the paper gets published. While they strive to eliminate the kind of subjective bias you get from small tightly knit communities (where subjective/invisible criteria are enforced in an informal way), they at least try to clear out the obvious junk. As a consequence, they too have become a half-repository, but at least one with a certain entry barrier. Then, at some point during the year, they make a collection of highlights or special picks for the previous year, which kinda work like what a conference would do.
> As a consequence, they too have become a half-repository, but at least one with a certain entry barrier.
eLife will still have an entry barrier, but passing that barrier doesn't give you access to publication, but to a positive rating in the assessment of the significance and rigour. In other words, it will still perform the function of highlighting potentially relevant research, but it doesn't block non-highlighted research from being accessible.
My experiences with PLoS Biology and PLoS Computational Biology have been great, but very similar to other journals.
Yes, it may seem wrong to put things into black and white buckets; but many our decisions inevitably are binary and we have to make them no matter how hard it is.
> with a short editorial assessment of the work’s significance and rigour
However, not getting the stamp of approval will no longer lead to the work not being accessible to people.
So does that mean that the only practical difference is that now when you list your papers for a promotion or tenure review you will write “eLlfe (approved)” instead of “eLife”? Then what is all the fuss about?
People had very mixed experiences with old-eLife’s editorial decisions, and some felt it was a bit clubby: papers from some labs sailed through to acceptance while similar work from others got editorially rejected (without review) for seemingly minor reasons. Their policy against requesting new experiments could certainly be used as a cudgel: an editor can just say you’re not convinced by a control and the paper’s DOA.
Thus, they are both uniquely positioned to make sweeping changes and have people doubt them.
2. The reviews use a controlled vocabulary, so I think you’d say eLife (excellent approach, major impact) or whatever the words are.
The fuss is about publication and approval being seen as linked in the academic hivemind. Even if you're able to see them as separate steps yourself, that doesn't mean that the academic world at large sees them as such, so if this fails to gain widespread acceptance, the reputation of eLife-the-journal might suffer, and having been published there will no longer look as good on your CV as it currently does.
I won't cry for them.
Would it? You'd think that a hiring manager would rather see a description of what the student accomplished, and with what level of skill, rather than a rather opaque letter grade.
Of course, that would make it tough to run "cattle car" courses with thousands of students being graded with multiple-choice tests. I don't think that would be a bad thing, either.
It will be easier with GPT providing the evaluation.
Yeah we have. It's called reading the paper and applying judgement.
But my HR department which made me write "publish X papers in top-tier journals (see list)" as a condition in my probation agreement most certainly does not.
And alas, as long as I'm under probation, their opinion carries far more weight in the debate than any moral argument you could come up with.
Thanks for clearly stating how journal prestige is harmful to academics and scientists.
The point was that utopian idealism that ignores reality on the ground is not as useful as you might think.
Your comment was as useful as me saying if I was president I'd give people free money.
Many important papers these days are interdisciplinary. Like, medical students doing bioinformatics and data analysis. But it’s impossible to be proficient in all the methods. So, many parts of these papers are impossible to assess for a wide audience, trust and faith in a journal quality becomes most important.
Perhaps some journals only let grade "A" papers pass, but sometimes that same journal has an editor who might let their buddy's article slip through. If one is part of the old boys club, this is a nice situation.
What eLife wants are signed grades from editors and reviewers. The editor is supposed to attach a brief summary in a few sentences summarizing the reviews. Hopefully one might bother to read a sentence or two to evaluate a paper instead of merely looking at the cover of the journal.
eLife's move here is basically a statement that the system is corrupt. Those wanting to fight the corruption are trying to increase transparency and reduce arbitrary decisions.
Wrt that: the NeurIPS conference did a test once, splitting the PC in two and having both review the same batch of 100 papers (and arrive independently at a decision). Roughly speaking (top of my head), the two sub-PCs agreed on the top 15% and the bottom 25%. That is: both recognised the obvious good stuff and the clear rejects.
For the remaining 60%, it basically was 50-50: one sub-PC would arrive at accept and the other at reject.
Which, to me, points out 2 things: (1) there is something very broken for about 60% of submissions... (2) but the process works as we want for the other 40%.
So I don't think accepting everything would be ideal, but I'm more than happy to try.
From a CS point of view, the problem is that in many communities, what is 'uncontroversially good' depends on subtleties only known to insiders who are willing to comply with the quirks of a community. And most communities are very defensive when it comes to shielding their top venues from relative outsiders, who are not sufficiently close to the inner circle of key players. As a consequence, getting the top-venue stamp (required to please the administration) becomes a social game. I think having a more open review process and stronger editorial control (vs. reviewer power) can mitigate this problem to some extent, for the following reasons: 1. When a paper is good, but upsets the subjective feelings of reviewers, the system should be implemented in a way that there is a high social cost for killing a paper because of subjective/cosmetic issues. Having venues where papers that are sent out to reviewers are most likely reasonable and should only be killed because of 'hard' flaws creates such a system. 2. When a paper is bad and the editors push it through, publishing details of the paper's editorial flow would impose social costs on the editors. (The reviewers, who may be potentially early in their career and may need protection, may even remain anonymous, the editors, who typically enjoy a strong standing in the community, should not.)
In CS, journals tend to operate a bit closer to this model than conferences, which I think makes the journal submission and revision process more meaningful. To conclude, I don't think getting rid of rejects is the solution, but rather editorial policies that encourage reviewers to either expose fatal technical flaws in a paper or help the authors improve the presentation to achieve a better end-result.
Humans have flaws that operate under the surface and it's good to have a system that minimizes the effect of ego on the dissemination of ideas.
I don't think the problem here is actually the "broken peer review system". I think this is a fairly natural development: if your PhD advisor insists that you have a few "good papers" before they'll agree to let you graduate, and "publication venue" is used as a proxy for "is a good paper" (which is very reasonable, because it might take years before better proxies such as "citation count" give a good signal), then you'll submit your paper to top tier venues and hope for the best. I don't really know how to fix this: Establishing things like TMLR (Transactions of Machine Learning Research -- a rolling-release journal meant to have roughly the quality of NeurIPS) might be a good way forward. Students get their stamp of approval by publishing there and NeurIPS can significantly increase their acceptance threshold. But once you do that, you'll risk that TMLR doesn't count as "good enough venue" anymore....
An important caveat is that this experience isn't accepting everything; it's publishing everything (or at least, everything that satisfies the desk check and is sent out for peer review). The whole point is separating evaluation from publication; they'll still distinguish works that they think are significant and rigorous from those they do not, but all works will be accessible. That's the mental shift they're trying to make happen.
(Disclosure: I volunteer for a project with a similar goal, that has received funding from eLife in the past: https://plaudit.pub.)
Anonymity obviously can't be provided when reviews or endorsements are public, but partiality can at least be detected more easily.
My conclusion is that there is an inherent subjectivity involved in judging paper "quality". Which shouldn't be that surprising; if there were objective criteria we probably wouldn't need humans to do any judging. I dunno if you want to call that a broken system or not, but if this outcome is unavoidable then trying to change it won't improve anything. (I suspect that 60% isn't some magic lower limit and improvement is possible, but I suspect the lower limit is 20-40% at minimum).
Realistically, this means that a single acceptance/rejection shouldn't decide the fate of a persons whole career. And they generally don't, it will be an average of many.
In a case like that, my advice to the editor is not a review, it is to simply reject for failing to follow guidance to authors. Now, it may be that other journals have a better desk review process before sending out to reviewers, but I don't know.
We already have blogs, arxiv, and similar. The main addition here seems to be that editors write something in addition to peer reviews. Where will they find the time to write these meta-reviews, and to deal with the inevitable complaints that may follow? This is asking a lot, particularly when it will be in service of a journal unlikely to have a good reputation.
Why wouldn’t tenure and grant committee members take into account high review scores in a journal that has high status and maintains an excellent review system? As long as eLife can maintain their status, maybe this could be a model for a positive change in science publishing, without making it any harder for people to get tenure, promotions, or grants?
Side note, it seems like this thinking that we need to have peer validation gate-keeping the publishing process is part of the reason for the reproducibility crisis; the reason nobody dares writing reproduction papers is because it’s so hard to get them published. Could help if journals accepted these by default?
I think that's also where their distinction between significance and rigour comes into play. Reproductions might not be "significant", but if they're rigorous, they can still get that stamp of approval.
> They worried it would diminish the prestige of a brand they’d worked hard to build
are disappointing to hear. eLife's good brand is a way to get their foot in the door, to change the system from within, but the dependence on good journal brands in the academic world is the primary reason that publishers are able to extort the academic world and lock the outcomes of publicly-funded research behind paywalls.
It's hard to not become the thing you set out to fight once you're in the system, but I appreciate that eLife and Eisen are still pushing to change it from within, and I hope they succeed, and that they overcome pushback from the vested interests.
https://www.science.org/content/article/rude-paper-reviews-a...