The fate of published peer-reviewed articles, submitted again (1982) | Hacker News Reader

The fate of published peer-reviewed articles, submitted again (1982)

journals.cambridge.org

53 pointsjacoblyles46 comments

This is an early paper by Stephen Ceci, and illustrates the approach Ceci takes of challenging the assumptions that other psychologists assume to be true. This is why I love reading books and articles by Ceci. His Web-based CV

http://far.human.cornell.edu/FAR/uploads/webcv/sjc9_webcv.pd...

lists his extensive publications (you hope that most of those got better peer review after his 1982 paper than psychology papers used to). Google Scholar

http://scholar.google.com/scholar?hl=en&q=Stephen+Ceci&#...

helpfully lists some papers by Ceci for which there are free full-text versions that you or I can download for our learning and reading pleasure.

From Jelte Wicherts (another psychologist skeptical about peer review in psychology) writing in Frontiers of Computational Neuroscience (an open-access journal) comes a much more recent set of general suggestions

Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020

http://www.frontiersin.org/Computational_Neuroscience/10.338...

on how to make the peer-review process in scientific publishing more reliable. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence.

"With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."

Two of my biggest gripes with the peer-review process would be relatively easy to address:

1. Time

2. One-way communication

1. You sometimes have to waste a lot of time to get articles published. The way the current process works, it is very easy for referees who are either competitors, incompetent, or simply assholes to effectively filibuster the review process. Even when things go smoothly the process is often agonizingly slow.

2. The peer-review process operates under the obsolete limitation of snail-mail correspondence. There is no two-way communication between the referees. Referees often return either unprofessional or obviously wrong (to everyone but the editor!) comments. If the referees could talk to each other this would be sorted out quickly. It doesn't help that editors are typically very reluctant to reconsider rejection verdicts even when they are based on clear cases of referee error.

How would I address this? Use the #$%@!ing internet people! Journals should abandon this one-way snail-mail process and build a private message board for each paper being reviewed. Referees are invited to post their comments anonymously on the forum. Other referees and, ideally, the authors could then reply to those comments. After a reasonable time period has elapsed, perhaps a couple of weeks, the editor should take stock of what valid concerns remain and reach a verdict.

Snail mail? I haven't seen this as part of the reviewing process in decades. What field and country are you talking about?

They use email, but I meant that the process hasn't really adapted to take advantage newer technology than snail-mail.

How so? The one time I was a referee I got the paper by downloading a PDF and sent in my comments by entering them into a form. That technology is newer than snail mail.

There are lots of things you might consider, that would be impossible with snail mail. For example, the reviewers might be able to send regular emails with questions to the authors, or the authors might be able to constantly update the paper in response to initial reviewer comments, there could be an anonymous "chat room" of some kind, etc. Not sure if any of these are good ideas, but I think it is a fair point that most journals simply use the internet as a "faster snail mail".

That said, I review for conferences that are taking advantage of the internet, with author response to reviewer comments, discussions among the reviewers, etc.

Regular emails with questions would certainly be possible with snail mail. But the fundamental problem with all of those suggestions is that only reviewers who really wanted to take a lot of time and care with reviewing would take advantage of them, and since there number of reviewers like that is vanishingly small, it wouldn't be worth the effort to establish the necessary systems.

I totally agree-- the problem with reviews isn't technological. Reviewers fundamentally need better incentives to do a good job, which I'm not sure how to do.

Certainly not a new problem (this paper is from 1982, so old that my University subscription doesn't cover it). This might be a better link to start such a discussion: http://jrs.sagepub.com/content/99/4/178.long

Interesting article, albeit with serious methodological flaws.

EDIT: Oops, I didn't see Cornell there, I take it back.

Your edit is itself very interesting :) Did the Cornell affiliation of one of the authors cause you to change your opinion in a positive or a negative way, and why?

Only joking. :) The joke being something along the lines of "look - I can peer review papers too - that wasn't so hard".

Buy the way, I'd love to see this research taken one step further, in a humoristic direction: a full statistical analysis comparing the predictive power of academic affiliation in determining acceptance for publication, to that of the contents of the paper itself. :P

There's just one solution. Make the reviews public. All academia I encounter are against it. So, it's probably the right way to go. :-)

The other popularly suggested solution is sort-of the opposite: use double-blind reviewing, where the reviewer doesn't know who the authors are, at least formally.

Two counter-arguments: First, in many disciplines, it is common to circulate pre-prints of papers which are being refereed. In this case, anonymity for the authors is impossible, practically speaking.

Second, the OP describes a snob effect, but there is also the reverse: Some referees will be more patient in dealing with badly organized or poorly written papers from a junior researcher. Not necessarily lower standards, but spending more time figuring out the paper, making suggestions for improvement.

And it can be obvious who the authors are in a field you are following closely; which, ideally, you should be if you are reviewing papers in that field. That's why I said "formally".

This paper extends our previous work on .....

This is the standard in most disciplines. I wish the authors of this article had done the same test with journals that do blind reviewing. I'd bet they still got a lot of rejections of previously accepted articles for bizarre reasons, since in general acceptance depends a lot on the particular reviewers you draw.

"This is the standard in most disciplines."

Any citation for this? I'm curious, because in physics, although this is often suggested, it has not been generally adopted.

Really?

I guess I should've said "this is standard in most disciplines ... that I know about". I'm really surprised that it isn't done in physics!

It's not done in CS either. It varies greatly by sub-topic though.

In computer vision conferences, reviewing is almost always double-blind.

In computer vision journals, reviewing is only half-duplex blind -- the reviewers can see who you are, but not vice versa.

> Make the reviews public

At least The European Geosciences Union publishes 15 academic journals that do exactly that. (They're also Open Access, and CC-licensed.)

http://www.egu.eu/publications/open-access-journals/

Some on them, like Atmospheric Chemistry and Physics, are also pretty highly regarded.

Here's hoping open, post-publication review catches on:

http://futureofscipub.wordpress.com/open-post-publication-pe... http://www.frontiersin.org/Computational_Neuroscience/10.338... http://retractionwatch.wordpress.com/2013/03/22/brian-deers-...

In academia, articles already have a form of post-publication review: citation count. If you don't do good work, you won't get any citations, though the converse isn't always true.

Professors usually ask their grad students to read the most important and influential papers in the field. When those grad students then turn around and start composing papers of their own, they'll refer to the ones they read, creating a kind of circular feedback cycle. Thus, if you want to find a field's best work, it often suffices to look for highly cited papers first.

It's like Reddit's karma voting system, just a lot slower. :)

>If you don't do good work, you won't get any citations

A lot of terrible, and even retracted papers have a high citation count, though.

True, but citations are qualitative as well as quantitative, so at least researchers should know that Controversial Paper X mostly gets citations as an example of "unusual findings, possibly due to methodological shortcomings" if they are deciding where it's worth summarising in the literature review section of their paper. And arguably (especially in social sciences) papers which provoke interesting rebuttals them are deservedly more influential than papers which are esoteric, trivial or dull.

Sure, just like how a lot of terrible and uninteresting articles on Reddit have a high karma count. By and large, both systems do a fairly good job at sorting out the interesting from the uninteresting, though there are always mistakes

If grad students are reading the most important and influential papers in the field then they're certainly not going to read the just published paper that tells everyone how to exploit magitech energy for great win - or whatever - because it's not important or influential yet.

All that process should do is reinforce the status-qua.

Post-publication peer review is already catching on, for example on Epistemio - http://www.epistemio.com.

Twelve years later, Caltech provost David Goodstein wrote http://www.its.caltech.edu/~dg/crunch_art.html Skip down to "honesty and ethical behavior" for his remarks on peer review.

I find it kind of funny that an article criticizing the current review practices is not free and openly available.

Of course it is. You just have to know where to look:

https://www.google.com/search?q=Richard+Smith+peer+review+fi...

Here's the .pdf: http://www2.psych.ubc.ca/~schaller/349and449Readings/Smith20...

In my field, it's common (and even encouraged) for authors to post their work to their own website. When authors give up their copyright to the IEEE, they explicitly retain the right to redistribute it on their own personal web sites.. From the text of the CVPR IEEE copyright agreement:

    Author Online Use
    6. Personal Servers. Authors and/or their employers shall have the right to post the accepted version of IEEE-copyrighted articles on their own
    personal servers or the servers of their institutions or employers without permission from IEEE, provided that the posted version includes a
    prominently displayed IEEE copyright notice and, when published, a full citation to the original IEEE publication, including a link to the article
    abstract in IEEE Xplore. Authors shall not post the final, published versions of their papers.

(the last sentence refers to the "camera-ready" version, which ideally only contains eg. typo fixes relative to the accepted version)

It's a different article though, which only briefly summarises the original post.

Worth a read though, for the author's recollection of his (unsuccessful) attempts to reject a Karl Popper paper he thought was substandard.

Open publication and review practices are nearly orthogonal issues.

At least one aspect of the peer review process should be better now than in 1982, when the paper was published: resubmission detection. Detecting plagiarism seems easy these days, just google for "detect plagiarism". But are reviewers actually using these tools? It would be interesting to repeat the resubmission experiment and see the results.

Absolutely.

Every reviewer that I've met does plagiarism detection on their own. My advisor has even found instances where the author copy+pasted articles from the likes of the Wikipedia article on the subject or 'about.com'; such papers get desk-rejected without review, and in some cases, the author is forbidden from submitting to IEEE conferences for the rest of their career. That's how important plagiarism is.

Just copy+pasting the abstract into Google is often enough to find it. At least for computer science conferences, the conference organizers run the papers through additional plagiarism screening tools before publishing, but it's mostly the reviewers' job.

"Every reviewer that I've met does plagiarism detection on their own."

I don't think it's a bad idea, but it never occurred to me to do this. If you're familiar with the field in which you are reviewing papers, you'll know if something is not new, in which case it is rejected, quite aside from the possibility of the author having lifted some actual text.

Sure, but plagiarism is more than just trying to publish an old idea. One paper was of something novel, but in their background section, they just copied a paragraph or two from an older article. That's the no-no.

You're right, my comment was not thoroughly considered. Maybe I should start doing some of this plagiarism detection. For some reason the idea depresses me, but you make a good case for it.

I just recently received a review of my paper, and it came with a pdf-report of which text fragments match a database of previous publications and the internet, made by some software called iThenticate. So clearly the journal now has a policy to automatically run this test for every submitted manuscript.

I haven't seen this before, so maybe that is getting more common these years now.

They do act actually. Plagiarized paper I found: http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=16...

Correction again. Apparently the IEEE takes action if I EXPLICITLY say which paper does copy which other paper. But here the same author is copying - including pictures - another paper again:

Search on Google scholar: http://scholar.google.com/scholar?hl=en&as_sdt=1,5&a...

Do I really have to write IEEE for all violations? http://ieeexplore.ieee.org/search/searchresult.jsp?searchWit...

It's just searching for certain phrases in the text. In this case the first one I tried "This process is defined as binding process. In The immune system" was already enough!

And it's sad.

Hard to detect plagiarism when the original article is behind a paywall of another publisher

Many publishers allow Google to crawl the full text of the protected resources. This is why Google Scholar works, for example. Copy+pasting the abstract into Google Scholar is surprisingly effective at finding plagiarism.

True, to some extend, but see the comment by gcr: https://news.ycombinator.com/item?id=5781899

Reply on news.ycombinator.com