Peer review: the end of an error?
the-tls.co.uk
the-tls.co.uk
https://en.wikipedia.org/wiki/The_Cost_of_Knowledge
He has practiced what he preaches with respect to publishing in open access journals.
A bit earlier, the famous anecdote is about Einstein's offence that the Phys Rev editor would dare to send his paper for review at all -- apparently this would not have happened at a German journal of that era: http://www.preposterousuniverse.com/blog/2005/09/16/einstein...
The problems are in some ways even worse in the humanities: http://www.chronicle.com/article/The-New-Intellectuals/23835... than the sciences. Or see https://jakeseliger.com/2013/02/12/a-lot-of-academic-researc... .
I'm afraid it doesn't ensure that all papers will be read. Some papers can attract more attention, some papers can even receive too much comments with relevant comments being drown in the sea of observations.
Research institutions that produce papers to be reviewed, pay up the reviewers, and then we make the research publicly available, because everyone is paid and happy. No need to double-pay the reviewers via per-paper costs etc. The cost of reviewing a paper is fixed. Hosting costs are trivial compared to that. The uni can take them on. In a sense sci-hub fixed the peer-reviewed industry. Formalize it (make all journals open-access) and we're done.
The solution is to offer a largish reward to the most constructive peer review for each manuscript chosen by the editor, or better by other reviewers. If I (or more likely one of my starving grad students) knew that I would get $1000 for providing the most constructive peer review then I would put in the required effort to make peer review a worthwhile exercise.
One thing that would help the peer review process is a data dump of the raw data. Many times I have seen some figure or table in a manuscript and felt I could not provide a constructive critic without seeing the underlying data. Going back through the editor to get this data is just too much of a hassle most of the time.
Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There's a lot of value, even if it's not perfect, in writers being forced to respond to one or two reviewers' comments.
The suggestion of going back and doing more work exists under the current peer review system. With my proposal you might even get suggestions of what work needs to be done and why you should do it.
In all the papers I have published I have only had a few reviewers agree on anything. The normal approach is you agree to all the reasonable demands even if you don’t personally like them and argue over the important one. Most of the time if the reviewers see you have moved some of the way towards their arbitrary demands they will let you hold onto the important ones.
I wasn't suggesting reviewers currently can't tell you to go back and do more work, quite the opposite - that I think it's a valuable if often too arbitrary requirement. But if the writer isn't picking which comments to respond to my point is moot anyway.
For a start, it would be good if as an author I could give feedback on review quality. Rebuttal periods are becoming standard for computer science conferences, and you can put some feedback into your rebuttal, but if you are too frank about bad reviewers, they could decide to recommend rejection. It would be good to add an option to review reviewers after the final decision on the paper has been made. I do think that that would make (some) reviewers more constructive.
A few conferences do have distinguished reviewer awards, though I've never really seen explained how the winners are chosen.
I'm afraid we're already there. Journals are very much strongly incentivized to publish only interesting or novel results.
There's a balance to be struck in how certain these things should be before being published, for sure. The p-hacking plague is a symptom of setting this certainty bar too low, or equivalently, the required level of novelty too high.
Things which stay constant are the job of standards bodies, and textbook writers... these are important too, but different.
Sometimes this isn't a big deal (medicine is as far as I know regulated enough that reproduction is a part of the problem), but most science is characterised by not enough money distributed without caring about scientific outcomes.
So we do have a problem at the moment. A team who does important but thankless scientific work can't get published. Too much science is just first draft status. And the only way to determine that is that people keep trying and trying to build on it and eventually decades later the tower of cards collapses when clever new PhD students start down new research avenues.
It will work for mathy and theoretical sciences like CS and theoretical physics, but good luck getting experimentalists to review papers for free (or at least without being asked by a journal). I'm not sure whether peer review is necessary in those cases, but the culture around more applied sciences definitely considers peer review more important, even though we routinely post to arxiv anyway.
Scientists at least in my field don't do each others' dirty work, we're much more disconected from each other than that. Also, peer review is anonymous.
1. New researchers. The examples he mentions include researchers who were already important in the field, and whose prestige guarantees review. But if you're a new researcher, good luck getting noticed. Blind review doesn't have this problem.
2. The cost of journals. People say "journal" and think of Elsevier, but there are several journals and conferences that publish all papers for free. The ACL, the largest conference on computational linguistics, finances itself with the entry fee to their conference, and you can read all of their papers right now (disclaimer: I'm currently doing work for them). There is a middle ground between Elsevier and Arxiv.
On that basis, maybe the mark of a truly prestigious researcher is one who can post directly to arXiv? Rather than the highest accolade being for a paper in Nature, the highest accolade should be for a widely reviewed paper on arXiv?
see also: "web of trust"
You are correct that it will take a long time to update documentation and policies that reference impact scores, but informally, at least in Machine Learning, people know who the contributors are with high actual impact. In addition because a lot of impactful work is coming out of industry and there is rapid turnover between academia and industry the culture as a whole is evolving.
If you've explored a problem, designed a piece of technology, and taken it back to that audience to gain further insights into the problem, what I can do in reviewing it isn't the same kind of analysis as a maths proof. Even if I you were to give me the tech or the code, I wouldn't have the same people to test it with, and would not expect to get exactly the same results. Instead, peer review tends to criticise the writing, missing elements in the argument, pieces of literature that also need considering, aspects of the methodology, and suggest more things you could do.
All of which is very valuable, and sometimes peer review is the best (only) mentoring you'll get if you're in a small university. But it's not quite the same as "we have to review this or a wrong claim might get the imprimatur of being science". Design questions don't make the same claims of completeness or context-independence. It's more of an argument based in the kind of muddy evidence you have to put up with when tackling muddy problems.
Which makes me think that instead of peer review being a hurdle to publication, perhaps we should just make the whole argument and refinement public. Give the reviewers more kudos as instead of being invisible entities whose thoughts are only received by the author and editor, their role in the development of that idea becomes a published item too.
Publishing in CS conferences/journals means you get reviews from 3-6 "smart people", and then maybe 50-100 people will read your paper, and most of those people will never contact you.
OTOH if you publish a blog and advertise on hackernews and twitter, you get instant feedback, including from some of the same experts that would have done reviews for conferences/journals.
Fortunately, the bad ones were rejected. The good ones ended up making me well-recognized in some extremely tiny niches, and ultimately did way more for my career than my PhD work did :D I'm really glad some of the bad ones didn't see the light of day :P
This is the opposite of reckless, this is how you make a name for yourself, and how you prove that you can develop an independent research agenda. As you say, it's good for your career, and it's good for science too (if the work is good, of course). Good for you!
I guess it depends on the field, but in mine (biological sciences) I can’t think of an example where a student would submit a paper without their supervisor knowing.
Anyone thinking of doing the same as you should try and draw on their colleagues in the department to proofread the manuscript. I used to proof lots of manuscripts from my colleagues and their students.
Of course I'm not proposing using SO directly for academic research but something inspired by it.
The focus should be on reproducing studies, not on peer-review.
Of course peer review can be of assistance in selecting studies worth putting the effort and funding into reproducing, but peer-review itself is extremely corrupted as a process, but since so many peoples careers depend on it , many of them will not talk about those issues.
I used to hold the peer-review and journal process on a pedestal. Then when working at a genetics lab I ended up having to read a lot of papers to get my sysadmin work done, and started realizing just how bad so many papers are, even though published in "reputable" journals.
For example, since publishing is seen as so vital to some people, I noticed many of them would "consult" on another groups paper just to get their name on the paper, even though they did almost no work, and then those same people would turn around and say stuff like "I've been published X times!" as a sort of reinforcement of authoritativeness (and as advertisement).
People are gaming the system because hardly anyone is actually checking them. Beyond that, the actual peer-review process itself seems to be very lackluster in rigor on it own.
I think there should be a certain amount of public research funding that should be earmarked to go directly to independent reproducability labs.
On top of it all, all this research that the public is funding is often behind a fucking paywall... and that's not just bullshit, it's theft and misuse of public funds. It should be required that publicly funded research should be publicly published, and therefore available for more eyes to see and review.
The two aren't separable, since a key part of peer review is assuring that a published study is, in principle, reproducible. You can't focus on reproduction withoit focussing on peer review.
But the fact that your peer (i.e. another academic who works in the same area) has read your paper and commented on it in the process of publication (i.e. customary "formal peer review") does not reflect on its reproducibility. The peer, being a human, has attentional deficits and there's probably true facts that increase the chances of the result occurring by chance which are not considered worthy of publication.
Until a paper's result is reproduced, it can't be regarded as reproducible.
I mean, isn't reproducing a study the ultimate peer-review?
If you are an academic or work in industrial research, it's part of your paid job to do peer review as part of the community. You are paid for it - it's part of your job and it's normal to do it during work hours.
A more salient point about why it is important to note that peer review is unpaid is that academics, as authors, submit their work---for free---to a journal, where reviewers review it for free, and then the journal turns around and sells the article for $35 apiece.
I see it as a community. We write and share our papers as part of our jobs, we review each others' papers as part of our jobs, we help to run the conference as part of our jobs. Everyone contributes to run the system.
Nobody buys articles for $35 a piece. All major universities and companies subscribe to the ACM or whatever your field's organisation is for you. And guess who runs the ACM and other organisations? The community does! People from the community serve as leadership and help run things.
Yes the ACM gets some money to run their website, conferences and outreach programs. I don't think that's crazy.
I don't mean to imply that peer review is this shitty thing that we "have" to do. I review papers regularly and generally enjoy the process. But I think it's important to point out that no, we are not being paid for it, and actually typically the incentives in our jobs mean that peer review is a "thing we have to stuff in there somewhere", and certainly not a primary task.
that's an unfortunate word.
As computer scientists, we have it pretty good.
Maybe you'd like to take a look at these members of the community, who show they care about the community by refusing to participate in journals for certain publishers: http://thecostofknowledge.com/ It even has a Wikipedia article: https://en.wikipedia.org/w/index.php?title=The_Cost_of_Knowl... And here's an example of one of them: https://www.scottaaronson.com/writings/journal.pdf
I work in an industry research lab in the bay, and while there isn't a requirement to review X papers a year, it is expected that you keep up with the literature, and actively engaging in peer review is one way to do so. A few of my colleagues are on the paper committee for our major academic conference, and they end up reviewing 40+ papers a year and using a week or so to take part in the committee meetings.
This is like saying that undergraduate students are unpaid for doing their homework.
It's part of their education to learn how to take part in the community by doing things like reviewing papers.
And this "part of their education to learn" keeps really cheap labor running science. Even postdocs are considered trainees, because they will always be learning because is part of their "job"
Most (all?) PhD students in science and engineering are on fellowships, at least in the US. Major research universities will not admit unfunded PhD students.
I was at a conference not too long ago and chatting with a fellow faculty member at another university, pretty well-known in his field (in a medical school department), who told me a story about how his chair berided him for doing peer reviews at all. The colleague was like "but this is part of our job as a scientist" and the chair's response was "but what account number is that coming out of", as in, "who is paying for that?" The chair proceeded to explain that unless the department was being paid for his peer reviewing time, he shouldn't be doing it.
He subsequently explained that he feels like he's checking out of academics per se and trying to shift to public outreach activities because he's tired of trying to make a case for research integrity for its own sake, as opposed to being a vehicle for bringing funds into the department.
Things in higher education are quickly moving toward a state (or have already moved toward a state) where it is implicitly assumed that your job as a researcher is to bring in money, not to do research. That is, if you are paid by the state, you are not being paid to do a job (research) you are being paid to bring in even more money. You're a salesperson or adsperson not a laborer. Grad students are the laborers. I strongly disagree with this model, but it's where we are.
I used to think that conversations like the one with my colleague were kind of fictional dystopian kinds of conversations until I actually had one. And he was dead serious.
Universities are quickly becoming places to rent space for research, basically.
And research funding is going towards big hotel, travel expenses, and conference fees.
> Grad students are the laborers
Making the business highly profitable as there is no need to hire a large number of professors where you can offer tuition waivers and cheap stipends to a larger number of people who wants to get another degree
I have actually been paid by some journals, but only a token amount, like $30/paper.