Any ideas for how the system might work differently? Could peer review learn anything from the open source community?
1. Acting as a filter for what research is worth reading.
2. Providing input to the author on how to improve their work.
3. Influence the author's professional evaluation.
The first is the most blunt instrument: when something is not considered worth reading, it doesn't get published at all. This is problematic in the cases where the peer reviewers misjudge, or when an article is evaluated on more than its academic merit, e.g. whether it's "groundbreaking" enough for a particular journal.
The second is useful, but made less so due to the power imbalance - an author cannot judge for themselves whether the feedback makes sense in the context of their work. This is especially problematic when someone has to take their work to multiple journals and/or gets conflicting feedback.
And that power imbalance is the result of the third point: it can be more important to the author to get that stamp of approval by the reviewers than to meaningfully contribute to science, because that's the only way they can stay in academia.
Ideally we'd split them up, but that does require aligning the academic community - which is a hard problem, especially considering there are strong vested interests in the status quo.
(Disclosure: I do volunteer for a project (https://plaudit.pub) aimed at splitting evaluation from publication, because I think it's an important issue.)
At first, I thought that Plaudit adds some post peer review platform, or discussion forum, for each paper. Similar to OpenReview but for any paper. I think this might actually be a great idea.
I think that https://fermatslibrary.com/librarian actually provides a similar functionality. But it has not seen much adoption.
(That said, a big challenge with citation counts is that they take a long time to accumulate.)
As I mentioned, I think the three functions should be separated. Although Plaudit by itself does not facilitate giving feedback, it does not prevent it either. Ideally, if someone comes across e.g. an error in an article, they provide that feedback to the author (e.g. through an email, of using Hypothes.is - a project that I think you might like). There's no need to only contribute to improving scientific literature through a formal journal-assisted peer review process.
And who knows: if the author incorporates that feedback, the giver of the feedback might decide to endorse that article using Plaudit :)
As a reviewer, I have no objections towards my reviews going public. However, I have often had to tell authors that their paper is dumb (not in those terms, obviously), and I'd rather keep those reviews private. Being rejected is hard enough without the entire world knowing how much your paper sucks.
Of course, there are terrible reviewers too. But IMHO that's better solved via better area chairs than with a Twitter mob. I would rather not being 4chan into the review process.
Peer review grew up in a world where there were many fewer people engaging in science than today. Typically an editor would know everyone in the world who had contributed to the field in the past, and would have enough time to understand the ideas of each new entrant to the field as they started to submit papers. It relied on personal connections and deep and thoughtful understanding.
That has changed just due to the scale of the scientific endeavor today, and is no longer possible in that form.
I don't know how you fix that. But it points to why peer review worked well at one time and became some kind of gold standard and why it's failing to be some kind of gold standard today.
I think you have to parse what was going right about the system that created peer review in order to have any hope of parsing how to replace it at scale for a world that's grown vastly larger in terms of sheer population numbers.
Many fields have developed work arounds to peer review in the form of working papers, preprints and conferences. That this is necessary suggests that peer review be burned to the ground and the ashes sown with salt.
https://www.stat.cmu.edu/~larry/Peer-Review.pdf
> The refereeing process is very noisy, time consuming and arbitrary. We should be dissem- inating our research as widely as possible. Instead, we let two or three referees stand in between our work and the rest of our field. I think that most people are so used to our system, that they reflexively defend it when it is criticized. The purpose of doing research is to create new knowledge. This knowledge is useless unless it is disseminated. Refereeing is an impediment to dissemination.
...
> Some will argue that refereeing provides quality control. This is an illusion. Plenty of bad papers get published and plenty of good papers get rejected. Many think that the stamp of approval by having a paper accepted by the refereeing process is crucial for maintaining the integrity of the field. This attitude treats a field as if it is a priesthood with a set of infallible, wise elders deciding what is good and what is bad. It is also like a guild, which protects itself by making it harder for outsiders to compete with insiders.
Maximizing signal to noise is a terrible goal to put so much weight on. We should be trying to maximize the growth of knowledge, not minimizing the number of wrong papers, which is what max(s/n) leads to.
Follow the editorial philosophy of Max Planck editing Annalen der Physik during the period when that was the pre-eminent journal in Physics during the most productive epoch for Physics research ever.
> To shun much more the reproach of having suppressed strange opinions than that of having been too gentle in evaluating them.
IMO, filtering out noise is a big part of maximizing the growth of knowledge. I cannot read every paper in my area that goes up on Arxiv, let alone every paper in related areas. That's why it's useful, albeit imperfect, to have gatekeeping peer-review organizations to force experts to look at papers and pick out the ones they think are most useful.
There is a lot of noise on Arxiv.
Gate keeping and review do not require anything like the current peer review system. The current peer review system retards the flow of knowledge, or it would if anyone relied on journals to know the state of the field.
Gate keeping can be done by editorial review. If you really want to you can have accept/reject only with no revise and resubmit cycle like Sociological Science. They aim for 30 day from submission to publication.
Refereeing wastes more than $50m a year in value in Economics alone. Burn the system to the ground.
> IZA DP No. 12866: Is Scholarly Refereeing Productive (at the Margin)?
> In economics many articles are subjected to multiple rounds of refereeing at the same journal, which generates time costs of referees alone of at least $50 million. This process leads to remarkably longer publication lags than in other social sciences. We examine whether repeated refereeing produces any benefits, using an experiment at one journal that allows authors to submit under an accept/reject (fast-track or not) or the usual regime. We evaluate the scholarly impacts of articles by their subsequent citation histories, holding constant their sub-fields, authors' demographics and prior citations, and other characteristics. There is no payoff to refereeing beyond the first round and no difference between accept/reject articles and others. This result holds accounting for authors' selectivity into the two regimes, which we model formally to generate an empirical selection equation. This latter is used to provide instrumental estimates of the effect of each regime on scholarly impact.
https://www.iza.org/publications/dp/12866/is-scholarly-refer...
I think the computer science publication model of a few conferences per year with ~3 month submit-to-decision timelines, based on 3-5 reviews per paper, is pretty good. But it's probably not appropriate for fields with deep contributions that take substantial time to verify, like math.
That logic probably made sense back when journals were the primary means of communicating scientific results. But the Internet has changed the situation drastically. Scientists don't need to wait for a journal to publish things; they just go to arxiv.org. Nor do they need to wait for a journal to filter things through peer review; they have plenty of other ways of judging papers for themselves.
So at this point I don't think we need a "system" at all for what peer review used to accomplish.
If we move to a system where everyone just publishes on e.g. Arxiv what happens is that people will just follow certain researchers that they deem good (you can already see that happening in some fields). So you have cabals developing who only read each others articles and it becomes very hard for new researchers to break into.
This is only true if (a) it filters properly, and (b) scientists actually pay attention to what it filters.
I'm not sure either of those is actually true now (though they might have been true in the past).
> So you have cabals developing who only read each others articles and it becomes very hard for new researchers to break into
If "cabals" that are hard to break into are an issue, it's not because of lack of peer review; it's because governments are now pretty much the sole source of funding for scientific research, so there is stiff competition for grants and many people will do whatever it takes to elbow out others, including forming cabals.
The way to fix that is to not have governments be the sole (or even the primary) source of funds for scientific research.
I used to be on a health list that was ostensibly looking for a "cure" or treatment for cystic fibrosis. They frequently posted links to abstracts of papers and discussed it on list and I had mixed feelings about that. I often didn't understand the medical terminology and it was just a blurb and yadda.
At some point, I talked on the phone with a guy with a PhD that I knew through an entirely unrelated email list. And he told me to not bother to read the abstract if I couldn't get hold of the full paper because it often said something different from what the paper said. I was relieved to hear this and not at all surprised, so I quit reading those abstracts and trying to pretend to participate in those discussions or whatever.
Some years later, a woman on that same list offered to co-author a paper with me and give me credit and yadda. And then she changed her mind and began trashing me on list and unsubscribed.
Her son died of CF (at the age of 17) years and years before and she was in search for some pill to make it all better, obsessed with finding a cure because of his death. I use a lifestyle based approach. She was a smoker. I think the implication of my ideas is that it's partly her fault her son died. And I think her guilt over that possibility is why she's obsessed with finding an answer, but it has to be an answer that somehow absolves her of guilt and doesn't in any way suggest it's her fault. It needs to do the opposite for her to be satisfied.
I was homeless at the time that she offered to write a paper with me. Once she kicked me to the curb, I didn't really have any place else to go.
Anyway, regardless of your process, you need some signals of some sort. I relied on the opinion of the guy with the PhD in part because he had a PhD. And I think my life has been better for that.
It's good to have a so-called "bullshit detector" but one of the ways we filter information is by relying on our understanding that some people can be believed even if we don't understand what they are saying because it is over our head.
First, while this might apply to lay people trying to evaluate scientific claims, it should not apply to scientists trying to evaluate scientific claims in their own field.
Second, peer review does not even pretend to be the kind of filter you are talking about. The fact that a paper has been peer reviewed is not evidence that it is correct or that its claims should be believed. Indeed, many peer reviewed papers later turn out to be wrong. (For example, all of the papers that were part of the "replication crisis" that got a lot of recent press were peer reviewed.) All peer review signals as a filter is that someone presumed to have relevant expertise thought the paper was worth publishing. I think recent experience has shown that this signal is virtually worthless now, though it might have had value in the past.
I'm "just a lay person" but I'm also someone solving a real serious scientific problem and I'm dependent on trusting that people with PhD's who answer questions of mine about cell biology or whatever can be trusted because, unlike me, they had the capacity to pursue a PhD. I simply can't. I'm too busy solving a scientific problem like my life depends upon it because it does.
And I'm sure this is a really stupid to comment to leave. I've written several drafts and I don't know a good way to engage your comment.
I primarily replied to the comment about "infinite time" because I sort of have "infinite time" in that I'm an unemployed bum with a bit of unearned income who also occasionally tries to earn a few bucks and whose primary focus is on understanding a scientific question so that I don't die and all that.
And as someone with "infinite time" to devote to reading articles and what not because reading those articles and what not is a higher priority than almost anything else since my life depends upon it, I find that I still need filtering mechanisms.
Now that I've dug my grave deeper, I shall endeavor to tear myself away from the computer and try to get a little shut eye.
Have a good evening.
If they do, then they're wrong, since peer review explicitly disclaims any indication that a paper is correct or should be believed. And this is supported by the fact that a large fraction of peer-reviewed papers turn out to be wrong.
> I'm also someone solving a real serious scientific problem and I'm dependent on trusting that people with PhD's who answer questions of mine about cell biology or whatever can be trusted because, unlike me, they had the capacity to pursue a PhD.
It depends on what questions you're asking them and what their basis is for their answers. If their basis is solid experimental knowledge of whatever aspect of cell biology you're asking about, then yes, their answers are probably trustworthy. If their basis is speculation about something we don't have much experimental knowledge of, then no, their answers are probably not trustworthy. In either case, you're not going to find that out by looking at whether what they're saying has been peer reviewed. You're going to find out by asking them how they know the things they're telling you and critically evaluating their answers.
> I'm too busy solving a scientific problem
> I sort of have "infinite time"
You can't be "too busy" if you have "infinite time". It's just a question of how you choose to employ your time. It would seem to me to be a better idea to employ it in becoming an expert yourself, so you can critically evaluate what other experts are saying, than to look for a shortcut filter, whether it's peer review or anything else. Asking other experts questions can certainly help with that; but it might be helpful to view that not as looking for answers from them, but as using what they say to help you build your own expertise.
I'm going to try one last time to clarify what I meant and then I am done here because this is going all kinds of weird places I never expected.
I'm medically handicapped. I have a very serious and deadly condition and failing to die of it takes all my time.
That fact has prevented me from finishing my BS in Environmental Resource Management, a major chosen to further my career goals. My interest in and need for information on cell biology is driven by the fact that I was diagnosed somewhat late in life with a very serious genetic disorder.
So I wouldn't be interested in pursuing such knowledge if I weren't medically handicapped. I would be a city planner or something, like I wanted to be.
The fact that my medical condition has stomped my life into the ground and made me unemployable is why I have a lot of time on my hands and I choose to prioritize learning knowledge pertinent to addressing my health issues under circumstances where, no, it's simply not possible for me to do something like pursue a PhD in biology.
And I am getting results, so your opinions about how I should spend my time aside, I'm fairly satisfied with the choices I am making (all things considered), though I wish I didn't have this personal challenge. I wish I were healthy and life had gone differently. But it didn't.
> it's simply not possible for me to do something like pursue a PhD in biology
Becoming an expert is not the same as getting a credential. It's perfectly possible to become an expert in whatever particular condition you personally have, without getting a PhD in biology, or indeed any relevant degree. Indeed, since you say you are getting results, I suspect you are becoming an expert in your particular condition, whether you planned to or not. My wife has a chronic illness and she has been forced to become an expert in it (with no relevant degrees at all) even though she would much rather have spent the time on other things.
> I am getting results
I'm glad you are; that's the most important thing.
I think that's a great question. Imagine papers with pull requests, issues, and additional experiments contributed from anyone with relevant experience. And imagine those experiments being open data and up for third-party verification before publication. That could be powerful.
* Research papers should be living documents that are properly versioned; updates should get reviewed again. This obviously means that incremental improvements to papers and reviewing need to result in something tangible for one's CV. The versioning makes sure one can still point to a specific revision of a paper. Of course, updates should consider the implications of changes carefully (as is the case for software). For example, I might get familiar with a paper over the course of a month, and then use it as the cornerstone of one of my own works. Then, I want that the paper is stable and does not constantly mess with important formal definitions et cetera. Otherwise, people won't be able to agree on the implications the paper has for the community.
* It might be better to make review and reviewers fully visible and move to a more interactive review process. For obvious reasons, practically "double-blind" review does not work, because if you're a visible member of your community and submit a paper, most people can guess that it's yours. If we would move from "one-shot" (or: one-shot plus rebuttal) reviews to more interactive, public, non-anonymous discussions, we would get rid of at least some of the social back-channeling that is only available to the bigshots in the community.
Of course, I don't think we will get there quickly.
- the quicker but less rigorous way: set things up like a forum like HackerNews, let the crowd-sourced upvote and comment system work its magic; make the karma/reputation points act as a proxy for the quality and relevance of the article, its authors, as well as the commenters.
- the slower way: let articles "bubble" up from smaller but more specialized workshops toward more generalized and prestigious conferences and journals. Have the technical committee recommend the best articles to those more reputable venues, along with whatever it is the authors need to work on to re-submit the extended version at that "better" venue. This is kind of already the case with workshops or conferences that have an associated Special Issue of a journal; I'm advocating for a generalization of the practice to all venues and more than a 2-level hierarchy. The reputation of the lower venues could be backpropagated by the number of papers they successfully "bubble" up, PageRank style, so that would be the incentive the committees would have in recommending their papers to the other venues.