ML researcher replies to request to review paper for closed journal

The earliest really large-scale instance of this occurred in 2001, when more than 30,000 researchers signed a letter boycotting any journal which didn't make articles open access within 6 months of publication. The boycott meant not reviewing for, editing for, or publishing in, any such journal. Signatories included many of the best-known scientists in the world. One of the authors of the boycott was Nobel Laureate Harold Varmus, who had just completed a term as Director of the US National Institutes of Health.

It's unclear how many of those researchers followed through on the boycott. Some did -- I've met many scientists who say they send letters similar to LeCun's. I've also met some who signed, but who now admit (often sheepishly) that they didn't follow through.

Text of the letter is here:

http://www.plos.org/about/what-is-plos/early-history/

The machine learning community has been pushing for open access for quite awhile.

In 2001, the 40 editors of Machine Learning (some of whom I was quite proud to know), resigned to start the JMLR with this letter: http://www.sigir.org/forum/F2001/sigirFall01Letters.html . Note also their citation of the similar actions taken by the editors of Logic Programming in 1999.

EDIT: btw, the JMLR is at http://jmlr.csail.mit.edu/papers/ . moar edit: http://jmlr.csail.mit.edu/mloss/ for relevant open source code.

It's worth noting that Cosma Shalizi has been doing this for about six years now, and it's becoming increasingly popular in certain areas of academia.

Background:

http://masi.cscs.lsa.umich.edu/~crshalizi/weblog/442.html

http://masi.cscs.lsa.umich.edu/~crshalizi/weblog/864.html

http://gowers.wordpress.com/2012/01/21/elsevier-my-part-in-i...

Terry Tao made a good point when he joined the boycott of Elsevier journals. He said he would stop submitting to their journals, and stop doing editorial work for them, but declined to stop reviewing papers.

The reason he gave is that while he is at a stage in his career where there is no pressure to publish in any particular set of journals and so can exercise discretion in where he publishes, he recognizes that some mathematicians, for reasons beyond their control, are unable to do so, and he does not wish to penalize them.

For fields where many of the high impact journals are not yet open access, I think Tao's approach makes sense. Publishing in high impact journals greatly helps the careers of young researchers.

In my opinion, Yann LeCun is one of the top celebrities in ML/Neural Net research. His voice really has weight.

I did a course this semester at university about bioinformatics. I was very impressed by their attitude to knowledge sharing. Almost every paper referenced by our professor and any citations in those papers that I used were published online for no charge. But even better than that, the data and code used to generate their results was shared too. Our professor noted that in the bioinformatics field a paper is considered pretty much worthless if the data isn't made easily available in one of a few archives.

I really hope the rest of acadaemia can follow through. It seems to be we might make more progress in all of the sciences if results are easily replicable and research is published to all.

This is a very forthright reply by the researcher asked to review the paper. Right now, reviewing submissions to scientific journals is anonymous, and not well rewarded. Jelte Wicherts, writing in Frontiers of Computational Neuroscience, (an open-access journal) suggests new procedures

Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020

http://www.frontiersin.org/Computational_Neuroscience/10.338...

for making the peer-review process in scientific publishing more rewarding and more reliable too. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence.

"With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."

> I have pledged to no longer do any volunteer work, including reviewing, for non-open access publications, which unfortunately includes pretty much every publication from [commercial-publisher].

The volunteer aspect of peer review is the worst part. How much is an expert's time worth? How much is a two-sentence review worth to an author? If it were paid on an hourly basis using an honor system, it would be fantastic for quality and consistency.

That closed journals ask for volunteer review makes it doubly bad. But I still believe paid review for open access journals would be a good thing. I know, it is an unrealistic fantasy.

Professors and researchers should start charging for time spent helping the closed journals at the very least.

If you think your reviews are valuable, charge accordingly. You provide value to the publishing process and should capture that value. You are providing the service of "expertise". If you provided that service to a lawyer, you would earn several hundred dollars an hour.

There's probably a business opportunity in creating a clearinghouse for reviewers, editors, etc. and managing the accounting.

Edit: Grammaring more goodly

This is a great first step. But for this to really make a difference to the current traditional publishers - every researcher would need to follow this example. The other problem is access to all the previously published papers. So my question is : Who owns the copyright to a paper published in a major academic journal? Is it the original authors / researchers? One would think so since they are the authors. Or do they grant exclusive rights to the journals. If the authors own the copyrights, what is to stop them from uploading their papers to one of these Open access sites?

I really don't want to derail the thread, but one of the people participating in that G+-thread (Mike Taylor) has written some very good articles at http://reprog.wordpress.com/ (and I really can't remember if I picked up the subscription here or elsewhere)

I would love it if somebody could register a domain and create a canned reply that we could copy paste when asked to review papers with appropriate links to all relevant open access projects.

Who is that guy saying that publishing in arxiv is career suicide? Why are there ML people playing paper chase at universities when they can quickly make a fortune in industry and retire to a life of academic leisure. Of all people, ML researchers should have noticed that the 21st century has arrived.