Except it needn't be. Reddit-like voting and commenting site for scientists would work just as well to vet papers and provide valuable feedback to authors with nearly zero overhead.
Probably not. There's a well-known essay that explains why karma/votes/likes work very well for easy tidbits like cat photos, gifs, and puns. (The stuff that takes less than 5 seconds to evaluate and requires near-zero effort to judge whether one "likes" it or not.) Unfortunately, I don't have a cite because I forgot who the author was.
Longer topics such as a 20000 word dense research paper with supporting data that requires at least 2 hours of study by a reader/reviewer doesn't work for a karma points system. The effort is too high to contribute significant # of votes. It's the same difficulty for other long form content such as manuscripts for novels/books.
Instead of karma voting, these works require curation that passes the "signal of approval" from one person to the next. With book manuscripts, curation is first done by the literary agents and then the acquisition editors. With scholarly papers, it's a somewhat analogous process (committee 1st pass then forwards to the peer reviewers). Each "approver" has a reputation which signals to the next person that it's ok to invest the hours into studying the work. This social signaling mechanism is also similar for startup entrepreneurs' business plans and VCs. VCs would rather be exposed to your business idea from a referral rather than an unsolicited cold email. All 3 domain examples of book manuscripts, academic research, and business plans share the same theme: people have limited time to wade through a vast sea of poor quality content.
I think people can conclude that flipping through 100 cat photos is different than slogging through 100 dense papers. Another example of this dynamic is the Stackoverflow voting system. There have been several meta discussions lamenting the fact that the "easy" questions attracts a high volume of activity (answers + votes) but the "hard" questions languishes with little attention. That's because the easy questions are more analogous to the low-effort cat photos.
Another commenter in the thread mentioned that github solved workflow for programmers and the same could be done for peer reviewing academic papers. This is misunderstanding the underlying issue.
Evaluating dense papers is not an "information technology workflow problem" that's waiting for collaboration software to solve. The bottleneck is human effort to properly read papers to judge its correctness and significance. In 1687, a human would read Newton's Principia Mathematica at ~200 words-per-minute and after several hours, maybe he'd conclude that "yes, F=ma looks reasonable here". It's 300 years later and we're still the same slow humans reading papers at ~200 wpm. It's worse now because there's a larger glut of bad papers to wade through than before.
Peer review of novels, scientific papers, business plans, is not solvable by a Github/Sharepoint/DropBox. The bottleneck isn't the copying of pdf or LaTeX files from one person to the next. It's the hours of human effort to judge the papers and signal to the next person to also expend hours of human effort.
The issue about publishers isn't the filtering. The real issue is that they they charge too much money for filtering. The high profits is misleading some people to think it's an easy problem solved by github or a reddit-like voting system.
You're attacking a straw man. I said reddit-like, not reddit-clone. There are plenty of obvious problems with reddit's voting system. So what?
What is scientific consensus if not a vote among some random sampling of scientists studying a field. What's left is to formalize this process in an open system so we're not beholden to the publishers and other parties with perverse incentives.
> Evaluating dense papers is not an "information technology workflow problem" that's waiting for collaboration software to solve. The bottleneck is human effort to properly read papers to judge its correctness and significance.
You're wrong. Firstly, it's often impossible to judge the significance of a result even soon after publishing.
Secondly, reviewers already donate their time to review papers, so the human effort is already there. What's needed is a well-recognized system encoding a publishing process, whereby some random selection of scientists obtain access to early drafts with a way to provide commentary and ratings on various metrics, with final publication after some quality consensus is reached among the random selection of reviewers (as a rough idea).
This is exactly an information technology problem.
Yes, and that's what I understood you to mean and a reddit-like voting system for papers is what I was talking about. (I even wrote "reddit-like voting system" as my last sentence in the post you replied to.)
>You're wrong. Firstly, it's often impossible to judge the significance of a result even soon after publishing.
I'm not talking about "significance" in relation to the whole of human history. I mean "significance" in the sense of a threshold for requesting other people to invest their time & effort to study the paper and also the subsequent subscribers to read.
>, reviewers already donate their time to review papers, so the human effort is already there.
Yes, but the key you're missing is that they donate their time because a respectable _somebody_ requested them to do so. The triggers for motivating that donated effort matters.
If I'm a working scientist working 50 hours a week, I'm not logging into a reddit-like voting system and investing hours to read (mostly bad) papers that have 0 votes so I can submit my 1 vote. No, somebody I respect (e.g. a journal editor) needs to ping me and say, "hey, this paper looks interesting, can you review it?" After I peer review it, I may get some quid-pro-quo such as free access to the journal's other papers and/or when it's my turn to submit a paper, the editor will consider forwarding it. So yes, I'm not "paid in cash" but I'm paid "in kind" with some intangibles.
That social dynamic is not replicable in a reddit-like voting system.
Therefore your following scenario of:
>whereby some random selection of scientists obtain access to early drafts
... is not realistic because scientists do not want to be part of random pool to be assigned random papers. That's contrary to basic human nature! Scientists want papers referred to them by another human and not an algorithm. It's not a technology problem, it's a human problem.
All the software suggestions to solve this from programmers including ideas such as a reddit-like system, or a github-like system, or a blockchain type system, etc ... they all misunderstand the culture and psychological motivations of publishing.
>What is scientific consensus if not a vote among some random sampling of scientists studying a field.
Yes, I can see that if you reduce the problem to the simple abstraction of "votes", it seems like building a software platform to formally capture votes is The Solution.
What your simplification leaves out is all the human signaling required before people render their votes. That's the human dynamic that collaboration software doesn't solve.
(In any case, an accepted published paper isn't "consensus" per se; it's just another piece of evidence for people to weigh. The paper just passed through enough human curation filters that may be worthwhile to read.)
I agree with you that it is difficult to improve on the peer review process. At the same time, several researchers have pointed out that publishers aren't really involved in the peer review process. It should be possible to coordinate journal publication through a software platform, replacing the publishers without affecting the peer reviewers or editors.
Yes, I agree employees at Elsevier aren't doing the actual peer review. However, they do critical filtering work before the any journal editors see the paper. See example from Elsevier employee Angela Kerr.[1]
The programmers who think they can revolutionize academic publishing by developing a new peer-review collaboration platform don't even realize that people like Angela Kerr exist. (See repeated disgust in this thread about "peer reviewers are unpaid". That's true but it also makes Ms Kerr's paid work invisible.)
In addition to basic copy-editing, she prevents crackpot articles such as "Darwin's Theory of Evolution is proven wrong by Trump" from reaching the journal editors and wasting their time. (She may inadvertently forward some bad articles but she has enough education to reject many of them outright to minimize the effort by the journal's scientists.)
Administrative people like her cost money. One would have to replicate her duties somehow in the new software platform. That's because the journal editors (scientists) are busy and they absolutely will not want to log into a new collaboration platform and stare at a long queue of unfiltered papers that everybody uploaded. Even though Angela doesn't do the "peer review" per se, the journals still want her to make a first pass cut. None of the ideas I've seen from programmers integrate her function into their model of what peer-review software should do.
With something like Stackoverflow, you have volunteers scrolling through the queue of questions and deleting/editing bad ones so their highly valued experts don't have to waste time with them. That's workable for "programming questions" because they are typically less than 100 words. Even with smaller workload of not reading 20000-word research papers, volunteers still get burned out and quit.
For highly prestigious papers like Nature and Cell that would attract a high volume of uploaded papers, it's very doubtful you'd have volunteers like Stackoverflow to slog through all the junk. Therefore, you'd have to staff up some copy editors and pay them money to do unpleasant work.
The problem is that Elsevier charges an insane amount of money for subscriptions (~$800 million in profits with 36% profit margin). Yes, a new system can charge much less money and be more efficient but you still have to do the work that Elsevier does. The journal editors don't want to do it. The potential software model of synthesizing ideas from reddit+github+dropbox+bittorrent+Sharepoint+blockchain ... don't account for Angela Kerr. Programmers think only in terms of distributing files like pdf and LaTeX. That's why they think academic publishing is a solved software problem and not a human effort problem.
>It should be possible to coordinate journal publication through a software platform, replacing the publishers without affecting the peer reviewers or editors.
The software platform can replace Elsevier in particular, but it can't replace the fundamental "work activity of a publisher". Angela Kerr (or whoever employs her) wants to be paid to do the unpleasant work that journal scientists don't want to do. And she doesn't want to be paid in reddit-like "karma votes" or github-like "stars". The new system can pay Ms Kerr less money than Elsevier (and hopefully pass those savings on to subscribing libraries) but let's be clear that the journal scientists absolutely do not want a new software platform without her.
[1] https://www.elsevier.com/connect/confessions-of-a-managing-e...
There is great need for a software nonprofit to enable scientists and editors to collaborate with minimal overhead, because even $2000/paper is too painful. The true marginal cost should be under $200/paper for scientific editing,[1] and at that price point, open access will overwhelmingly win. If we can build such a platform, I believe that Elsevier will find it very difficult to retain their journal editors.