It’s time to allow researchers to submit manuscripts to multiple journals
nature.com
nature.com
A less spirited method is to let traditional publishing slowly die by adding review features to highly viable alternatives like arXiv, PubMed and others which will then replace it. I hate academic publishers, they are vultures that serve no purpose. I hope they all go out of business or somehow pay big :)
The article isn't about open access though, so this isn't strictly on topic.
[0] https://www.ukri.org/manage-your-award/publishing-your-resea...
[1] https://www.nihr.ac.uk/about-us/who-we-are/our-policies-and-...
You are already allowed to (at least in tech) provide a preprint and public PDF on personal (non commercial) or educational web sites.
And Google scholar does associate those with the publishers links, as an alternative source.
So, it's not like we couldn't provide all R&D for free already. At least in tech.
https://en.wikipedia.org/wiki/United_States_v._Swartz
He helped Assange who is now facing the same fate. And "the west" loves to point fingers that countries for their flawed justices systems. Hypocrisy to the max.
I wish Aaron Swartz had not committed suicide ahead of his many charges. I wish he had had his day in court to argue the illegitimacy of those laws in these cases in the first place. It wouldn't be the first time that produces a change in laws. We will never know, and must change the laws asap.
A system that seeks to inflict all the penalty it can is a system that is doing harm; it has fully lost site of it's original purpose to promote a specific good. Having lost it's way, it is fixating on fairness and outputting damage.
Laws created+pushed by Joe that put many in jail for drugs, were violated by Hunter, and he walks.
Law is a fluid concept in this day and age. Same like this "rules based international order".
Please remember that hiding "undersireables" in WW2 Germany was against the law, or freeing slaves was against the law.
This is a naively optimistic view of how knowledge production actually operates. Sure, the scientific endeavor is constrained by what is actually the case (i.e.reality), but without some kind of editorial oversight imposed from above nothing coherent nor useful will be produced. A thousand well-trained, well-intentioned researchers toiling away at a problem will not magically self-organize such that their collective efforts make tangible progress on large, intractable problems. This will be true regardless of how many fancy web2.0/3.0 niceties one throws at the problem, since experience has shown that such solutions only make the problems of social dominance worse, not better. In the end, this sentiment is nearly identical to people complaining about "capitalists".
Do capitalists and academic publishers have purposes to fulfill? Yes. Do they fulfill that purpose well these days? Absolutely not. Like many of our social institutions these days, the people who run them seem to fundamentally misunderstand what their roles are, deferring to some vague, financialized liberalism that believes all problems can be addressed by maximizing human-freedom, with no regard to bootstrapping problems. Because the institution ceases to perform it's role, people begin to believe it has no role. Worse yet, now that people have no idea what the institution's role even is, they have even less of a clue as to how to fix it.
True, but academic publishers charge an absurd amount of money in return for very little value. The publisher provides a platform for "editorial oversight" by peer reviewers, but they do not pay the peer reviewers. I would argue that "editorial oversight" in the form of peer review may be worth thousands of dollars per publication, simply providing a platform for that review and profiting from volunteer work should not be compensated as highly as it is right now.
1) Verifying that work is correct, assuming that the author is honest (e.g., you take their data at face value)
2) Verifying that work is correct, assuming that the author is malicious (e.g., you scrutinize their data to see if it's fabricated)
3) Certifying that the paper is "interesting" (universities, grant-making bodies, and other bureaucratic entities want some evidence that the researcher their funding is good, and somebody has to hand out the gold stars)
It takes time for even an expert to do 1), and it takes still more time to do 2). There aren't really good incentives to do it beyond caring about your field, or wanting to build on the thing you're reading. 3) can be done more quickly, but it's subjective, but a world where things are only assessed for correctness and not interesting-ness is a world where external funding bodies rely on other weird proxies like citation metrics or something to figure out who's good, and it's not clear to me that that's better.
My perception from computer science is that it should be harder to submit papers, because there are too many authors who simply rarely produce good papers and are clogging up the conferences with endless resubmissions until they get reviewers lazy enough to all say "weak accept".
It seems like the root issue here is pathological incentive to publish for career advancement?
This establishes priority (if credit is a concern), can be made blind / double-blind if so desired, and also makes public the reviews (which are as much a public service as writing research papers). Which editorial boards “accept” the paper for publication is then simply a matter of collecting endorsement tags on the submission.
It is quite well regarded now, they publish excellent research and the whole process is crystal clear.
So doing away with peer review would have some negative consequences.
It's also worth remembering that journals are really not very different to newspapers, and the top journals are the tabloids. They want to publish things that people will read and cite. They only care about quality in so far as it is required for people to want to read a paper. Ever wandered why so many top journal papers get retracted? It's because the top journals know people will read the click-bait! My favourite example is when the Lancet published the famous MMR-Autism paper by Wakefield, which is terribly unscientific, but was bound to be much cited.
the really interesting part is that sometimes review benefits from anonymity and even blinding. doing that in a more-public publishing process would be interesting. (but not impossible - there are tricks to providing authentic anonymous identities.)
But the parallel breaks, as unlike with code reviews, there is no protection against (and the people below are real people I have encountered in my 10+ year research experience):
* A reviewer writing eloquent reviews on every single paper they ever reviewed, explaining why they should be rejected, as a gimmick at this point.
* Reviewers not taking time constraints into consideration. Doing this 2 year experiment would indeed make for a better paper.
* Reviewers not reviewing in a timely manner. Yes, you said you volunteered to review this. It has been 9 months since.
* Reviewers requesting you to cite their papers, however unrelated.
* I quote a full review I once had: "This paper needs more equations."
When code reviewing in a team, these issues get sorted out. In academia, peer reviewing is in a state of complete anarchy.
Maybe peer review should be time limited, but have the reviewers comments be immediately available for the authors to respond to, rather than having to wait for them all to come in.
For example, in continual deep learning people often use time datasets in which they use a small amount of memory and incrementally learn classes and the algorithms cannot work for other incremental learning distributions. It's been very hard to publish work that instead works well for arbitrary multiple distributions, eliminates the memory constraint (which doesn't matter in the real world mostly), and shows it scales to real datasets. We have been able to get huge reductions in training time with no loss in predictive ability, but can't seem to get any of these papers published because the community says it is too unorthodox. It is far more efficient than periodically retraining as done in industry, which is what industry folks always tell me is the application they want from continual learning.
The confusing thing is that when I give talks or serve on panels I always have folks thank me and tell me they think this is the right direction and it was inspiring.
In my field the review system is way overtaxed with too many inexperienced people who struggle with having a broad perspective, so I think submitting to more venues would probably make things worse.
In general, the whole deep learning field tends to oversimplify practical problems and experimental design in favor of overcomplicating methods. This happens at all the levels.
* papers are generally posted to arxiv.org immediately on being finished, so everyone can access them
* conferences have fixed deadlines and relatively short, fixed/enforced review cyclesEveryone accessing someone's uncritically reviewed work is a bittersweet gift.
Math papers tend to be highly technical, read by other specialists in the field. When it comes for correctness -- whether or not I should take a paper with a grain of salt -- the authors' reputation counts for much more than the journal's. And in case of student authors, who are just beginning to publish, the advisor is implicitly staking their reputation on the work as well.
There are also preprints on the arXiv, written by people unknown in the community, claiming to prove the Riemann Hypothesis or some such. These aren't taken seriously by anyone.
An outsider might not be able to tell which preprints can be considered equivalent to papers, but such people are not likely to be seriously reading math research in the first place.
The informal one you describe here, or any formal one you can come up with.
Instead, think of the goal being to associate measures of worth with the reviewers. If you're publicly rated by a bunch of worthwhile people, count yourself lucky.
Exactly, that's why I am not suggesting any change to Arxiv.
Think more of people eg submitting Arxiv URLs to Hacker News for what I have in mind. Or discussing Arxiv submission on a forum or in a wiki etc. You can imagine some specialised software that has some better support specifically for material from Arxiv.
That's what I mean by 'overlay'.
Or think of Slatestarcodex publishing a blog post with links to his favourite Arxiv papers for that month. That's pretty much equivalent to what a journal produces. And if Slatestarcodex compiles that link list by doing some peer review and chatting with the authors of the papers, that's almost exactly what the journal does.
Review work is not always done by senior researcher (e.g., professors). Senior researchers often hand this down to PhDs. Having 3 to 4 reviews by nice junior reviewers doesn't sound very critical.
So PhD students writing the initial review is not weird - it is an expected part of their training. As is the supervisor going over the review and providing constructive feedback. As is the review being submitted under the supervisor's responsibility, with credits (mention in proceedings) to the student for acting as a subreviewer.
Yes, there are ways to abuse this system and yes, abuses do occur. Any system for gaining job prestige or workload reduction is a target for gaming. This doesn't mean the system should be thrashed, but it does warrant additions to curb excesses.
The sibling comments about making sure that most reviews are written by senior researchers also make good points. That should be checked by the program committee or editor.
It is of course true that published papers have been vetted. But very often, it simply means that 1. an editor glanced at it, 2. (optional) a peer provided a positive quick opinion on the paper, without checking the correctness, 3. one or two independent referees presumably read the paper and produced a report on it. It's not nothing, but it doesn't mean you should accept blindly as truth everything published.
For context, I'm an associate professor of mathematics at a large research university.
Peer review is the first filter that papers go through. It's not perfect (it makes mistakes in both directions), but the output of the peer review process definitely has a higher signal to noise ratio than the input.
https://browse.arxiv.org/pdf/2310.02335.pdf
Do not trust arxiv papers. They have not been vetted.
One thing that struck me as a researcher back in the day was that it was much easier to get people reading things that I put on my blog than it was getting scientists to read my papers. Basically as a researcher, you are basically engaging in 17th century SEO by writing papers and getting your peers to pay attention to this. We use email these days but otherwise the process hasn't changed a lot.
This is weird considering we now have a lot more tools. Imagine if hacker news worked that way. It wouldn't work. The best kind of endorsement for a paper is not random anonymous peer reviewers giving their thumbs up or down as to the inclusion of a digital only journal that nobody ever reads cover to cover. Instead it is other researchers citing your work. This is in fact part of how most academic performance is measured. The goal is to get your peers being aware of the existence of your work, get them to spend time reading or learning about it, and then getting them to engage with it by citing, criticizing, or adding to it.
The whole business of a journal publication is just SEO. You get some renowned journal to include you and maybe people will bother reading it and maybe refer your work if they like it. A citation is just a glorified like. The real goal is to get people to read and "like" your work. It's also self re-enforcing: the more people cite your work, the more people will read it and thus cite it. But it's a super inefficient process. From idea to citations happening can take years.
Conferences and workshops are where scientists meet up and discuss their work. That's where your peers are. I always enjoyed the smaller workshops. Get some smart people in a room and beautiful things happen. The academic world is basically a social network. With likes and everything. But minus all the modern tools that make other social networks work so well. There's some room for improvement.
But it may be worse than that, actually. I will not be much inclined to bother reviewing, if I know that the authors might pull their manuscript if another journal gives a green light quicker than the journal for which I have been asked to review.
The solution to a slow reviewing process is not to ask reviewers to do more of this unrewarded work.
Actually, here's a paper showing that the peer review is already overstretched:
The biggest problem I see with the current system is that a small number of authors venue shop by repeatedly submitting the same paper without changes.
Allowing submissions to happen in parallel will benefit that group of bad actors to the detriment to pretty much everyone else.
Also, all submitting in parallel will do is increase the number of erroneously accepted submissions. For one thing, typical reviewers review for multiple conferences, so you'd just be sending the same manuscript to the same person multiple times. Also, most related conferences have a fairly similar bar.
In fact, most papers have pretty good alignment on reviews. Reviewers typically can't see each other's reviews or discuss the paper before submitting their review, so these are mostly independent samples. There are papers that have bimodal scores, but it's rare, and usually due to low confidence reviews or the very occasional controversial paper.
It is often detected when people just ignore previous reviews, and resubmit without revision. Typically, one reviewer will overlap between the two conferences and notice the resubmission. If the paper hasn't changed, that's an easy reject, even if the person wasn't assigned to review it. Allowing parallel submissions would normalize this class of denial of service attack against reviewers.
Also, far more often, the previous reviewer will take a look, and say that the authors did a good job addressing the previous group's concerns. Of course, that can't happen if the paper is submitted in parallel. Allowing parallel submissions would put people that actually improve their work between submissions at an unfair disadvantage.
[1] https://www.nature.com/nature-portfolio/editorial-policies/p...
https://www.thelancet.com/journals/lancet/article/PIIS0140-6...
One half-baked idea: For academic hiring, only ever judge the quality of a candidate's research based on their five best papers (as they themselves have nominated), then there is no pressure to publish anything that doesn't break into the top five.
I mean, the idea looks nice, but there will always be people trying to game the system.
But don't get me wrong – I still like your idea, I just think it would need some refining (as you yourself admit, ofc).
* As in, no research.
in reality, faculty do more than publish, and are also judged by service, supervision, etc.
PhD students should probably spend much of their effort in trying to replicate published research papers, instead of publishing "original" research. This would teach them a lot about the typical quality issues in their research field, allowing them to produce quality "original" papers later.
This may sometimes even allow them to publish highly influencial papers that show the limitations of published papers, because quality issues seem so widely spread. This would also allow them to really contribute to the field.
I think we would see this if Universities and journals would take the research quality seriously.
modest proposal: always publish to arxiv, which assigns the permanent DOI. journals can be offered the article - one at a time if the authors desire, or in more of an auction format. when published in a journal, the arxiv entry is simply updated (new revision findable and attached to the original DOI).
this would make research works first-class, rather than journal byproducts.
internet/telecom providers don't want to be dumb-fat-pipes; journals don't want to be editing and review-coordination services. so what?
I'm all for putting camera ready copies on it, but submitted drafts really shouldn't be there.
I'm currently building a platform that aims to make these improvements and enable these experiments - the working title is "JournalHub". The one liner might be "Github for Journals". It's in alpha and currently supports preprints and preprint review, journal publishing flows, and post-publication review. I've done a bunch of iteration on the review UX, which draws heavily on Google Docs and Github PRs, and that's getting pretty close to beta. I'm still working on the editorial workflows which are in early alpha.
Once we have a platform that a significant number of journals are using, we can then build opt-in experiments with peer review. An example particularly relevant to the OP is potentially allowing multiple submissions where each editorial and review team can see the other's comments and thus multiple journals can collaboratively review a paper and benefit from each other's work. I've talked to editors who actually suggested the idea while doing UX research, so it's not unfathomable that journals might opt-in to something like that.
My impression from UX research is that there are actually a lot of editorial teams aware of the problems with journal publishing and eager to experiment, but feeling restricted by the mandates of the commercial publishers. So my main goal right now is to enable editorial teams to escape the commercial publishers, to free their hands, and enable their experiments.
I'm still doing UX research and looking for editors and editorial teams interested in talking - so if you're a journal editor and you want to help develop a platform that might help you escape the commercial publishers, reach out! (Email: dbingham@theroadgoeson.com)
Then over here we see a journal publishing this trash that treats their audience as toddlers with amazing tips like
"Follow up: after submission, "
"Professional networks:.."
Mostly I'm just surprised that our smartest people put up with this, when any researcher could ask their CS colleagues to create a replacement over a few weeks. It's a pure social coordination problem, it would only take a few groups of researchers switching to light the fire. It's a preference cascade just waiting for a little nudge.
(Of course, in 2023 you should be able to publish your data and all your software with the paper.)
- Peer review in reputable non-profit journals actually provides constructive suggestions that make papers and research itself better. APS's PRX and PRL, as well as Quantum are journals where I have seen these idealistic positive effects;
- Filtering out the obvious red flags is pretty valuable even if boring;
- Thanks to people who care about the "ideal" of peer review we now have the infrastructure necessary to make reproducability much easier: mandatory data and code sharing on archival services, open (even crowdsourced) peer review, immediate feedback, etc.
And while all you say is true, it's probably the most we can get for free in a reasonable amount of time; requiring an independent lab to repeat an experiment would generally be far more delay and cost than we'd accept, other researchers do generally want to see the outcome as soon as the first experiment is documented; and there are people doing great research which won't bother to submit if they'd have to pay for the replication - it's generally the bad research that has motivation to spend more money for a publication. The general public might want to wait for extra confirmation, but they're not the target audience of research papers, those are intended as communication by researchers for researchers. And also quite a few media outlets would disagree and probably prefer grabbing up hot rumors even earlier, even if they turn out to be false afterwards.
The truth is that good science is slow and that most “evidence-based” practices are referring to a huge, nebulous cloud of bad results and weak suggestions rather than the evidence that supposedly gives them authority over traditional or intuitive practices.
Scientists participate on “Little Science” and the responsible ones often maintain the perspective that you’re describing here.
But modern society has built itself around the institution of “Big Science” which is structurally forced to assert truths before they can responsibly be defended.
It’s way bigger than the general public being curious or the media wanting to get eyeballs — it’s everything going on in government, economics, medicine, psychology, agriculture, etc etc etc
It’s a house of cards and you’ve just summarized what the core problem is.
You can disagree with this approach, but then there needs to be huge budgets set aside for reproduction.
Sadly, integrity offices will rarely conduct serious investigations, and won't conclude misconduct happened unless what was done was incredibly harmful. Professors are often too big to fail, they attract tons of grants and are politically entrenched.
This is a nice sentiment but demonstrably false.
Fraud is common in academia and everyone knows it. A large part of academic science is a grift for funding. Is not "Trust" that is fundamental, is tit-for-tat.
- question whether or not the conclusions you are making are supported by the data you are presenting
- ask for additional experiments
- evaluate whether or not your research is sufficiently novel and properly contextualized
- spotting obvious red flags - you seem to discount this, but it's quite valuable
In my experience, the process of peer review has been onerous, sometimes taking years of work and many experiments, and has by and large led to a better end-product. There are not so great aspects of peer review, but it's definitely not a joke as you characterize it.
I'll add that in biology and adjacent fields, it makes no sense to discount peer review because the reviewers do not repeat your experiment - doing so is simply not practical, and you don't have to stretch your imagination very far to understand why.
This would massively increase the number of people that review a manuscript while also shortening the feedback cycle. The papers I've published have typically been in the peer review process for months to years with just a handful of feedback cycles of sometimes dubious utility. This can be improved!
Edit: I forgot to mention the issue of politics in peer review! If you're in a relatively small field, most of the big researchers all know each other, so peer review isn't truly blinded in practice. Junior researchers are also pressured into acquiescing to the peer reviewers rather than having an actual scientific debate (speaking from experience).
I pivoted away from attempting a crowd source review approach with a reputation system to trying to support journals in going Diamond Open Access.
But the platform I've built supports co-author collaboration, preprints and preprint review, journal publishing flows, and post publication review - all in a continuous flow that utilizes an interface drawing from Github PRs and Google Docs.
You can submit a paper, collect feedback from co-authors, then submit it as a preprint and collect preprint feedback, then submit to a journal and run the journal review process, then collect feedback on the final published paper. And you can manage multiple versions of the paper, collecting review rounds on each version, through that whole process.
It's in alpha, I'm pushing really hard with a short runway to get the journal flows to usable beta while trying to raise seed funding... the catch being I feel very strongly that it needs to be non-profit, so seed funding here is grants and donations.
I'm looking for journal editors who want to participate in UX research. I'm also interested in talking to folks who run preprint servers to see if they'd have any interest in using the platform. If you (being any reader) know any, or have leads for funding, reach out: dbingham@theroadgoeson.com
However, I think there's a lot that software can do to nudge towards better peer review. And once we have journals using a platform we can build lots of experimental features and make them easy to use and adopt to work towards improving it.
I've kept crowd sourcing preprint review in the platform - though I removed the reputation system since UX research suggested it was an active deterrent to people using the platform - to enable continued experimentation with it. And the platform makes it easy for preprint review to flow naturally into journal review and for the two to live comfortably alongside each other. The idea being that this should help enable people to experiment with preprint review with out having to take a risk by giving up journal publishing.
And the platform has crowdsourced post-publication review as well.
My thought is that if we can get the journals using the platform, that will get authors and reviewers in the platform and since preprint and post-publish review are really easy to do in the platform that will drastically increase the usage of both forms of review. Then folks can do metascience on all of the above and compare the three forms to see which is most effective. Hopefully that can then spur movement to better review.
I also want to do work to ensure all the artifacts (data, supplementary material, etc) of the paper live alongside it and are easily accessed during review. And work to better encourage, rewards, and recognize replications. I think there's a lot we can explore once we have a large portion of the scholarly community using a single platform.
The trick is getting there.
And then on the other hand, if you ever actually have to submit a paper to peer review, you'll see how clueless a lot of the reviewers actually are. About half do give useful critiques and comments, but the other half seem to have weird beliefs about the subject in question, and they pan your paper due to you not sharing said weird beliefs.
Yes. When I was teaching at the graduate school level, doctoral students sometimes came to me for advice about how they should respond to peer reviewer comments. Those comments were usually constructive and worthwhile, but sometimes they seemed to indicate either a misunderstanding or an ideological bias on the part of the reviewer. (This was in the social sciences, where ideology comes with the territory.) But even in those latter cases, the junior researchers just wanted to know how they could best placate the reviewer and get their paper published. None had the nerve, time, or desire for an actual scholarly debate.
There's no promise that an unreviewed pre-print is going to get two constructive readers. It's also wildly subject to bias - being on a pre-print with a junior, female researcher was eye opening as to the merits of double blind review.
You'd set up a server where people have accounts, but publishing pre-prints is anonymous by default, and identities can be revealed later.
In the current peer review system, people already have to produce papers with those identifiers removed. They can do exactly the same in the pre-print world, can't they?
That sort of betrays the idea of a pre-print, in my opinion, because they should not depend on "Someday we'll come back and fix this".
Edit: Also, to comment more on my own experience, I was lucky to be working in a well-established lab with a PI whose name carried a lot of weight and who had a lot of experience getting papers through the review process. We also had the resources to address requests that might've been too much for a less well-funded lab. I'm aware that this colours my views and didn't mean to suggest that peer review, or the publication process, are perfect. The main reason I wanted to provide my perspective is that I feel that on HN there's often an undercurrent of criticism that is levied against the state of scientific research that isn't entirely fair in ways that may not be obvious to readers that haven't experienced it first-hand.
I don't if this is how it's done already. I have seen people complaining about peer review here and was wondering why there isn't a solution to that while software already enjoys a well established peer review system.
if those were the worst problems with peer review, we'd be in a much better place. Your peer reviewers are frequently higher status scientists working (competing) in the same research area you are trying to publish in. Generally, they do not want their own work outshined or overthrown.
If I really hate them, I give them a "Major Revision" aka a laundry list of expensive follow-up experiments and then reject them on the 2nd or 3rd round after a few months.
There's actually zero benefit to earnestly engaging in peer review.
(I know it from personal experience).
Personally, I decided to leave and make a more honest living. It seems you chose not to.
When papers are reviewed, there are going to be a finite number of spots in the journal or conference to be assigned competitively. In good places, the reviewers catch issues in the papers and it won’t be easy to pass them.
Without peer review, a PhD student requesting graduation or a candidate applying for a faculty position would claim they have done major work, and there is no way to filter out the noise.