Stretching the “peer reviewed” brand until it snaps | Hacker News Reader

Stretching the “peer reviewed” brand until it snaps

blog.dshr.org

16 pointspadraic7a9 comments

It's interesting that the elite journals are more likely to issue retractions. Though I suppose the increased prestige and job opportunities logically lead to more motive to game the system, create sockpuppets, commit other fraud etc.

Does the high rate of retractions show that the system works to some extent though? I wonder what percentage of fraudulent research or faked peer review is actually exposed? I guess we'll never know - one more argument for open access and access to open research data though!

If the cost (in terms of reputation) of printing a paper and subsequently retracting it is lower than the cost of rejecting the paper in the first place, a journal is actually better off printing a dodgy paper and then backtracking later. Ultimately these journals are money-making functions of a business enterprise so they'll usually do what they can to maximise profit rather than quality, apparently even if that hinders the science.

although I think it's fair to say that not all retractions are the result of fraud. I'm not sure about the ins and outs of this case for example http://retractionwatch.com/2014/12/26/nature-cell-biology-in... but I think it looks like a straightforward revision due to more up to date information. This seems to be a good example of how science should work i.e. scientists posit theory, scientists receive new information and update or revise theory in light of this.

do you think this is just a by product of elite journals being targeted more often, rather than a conscious effort to optimise profit?

The more elite the journal, the more "important" it's likely a paper in it is. If the paper is truly important, people will try to build upon it, and in the process discover problems with it.

Contrast to "less elite" journals which in part serve the "publish or perish" market, and are therefore less likely to have material other people care about. The smarter cheaters will obviously game that general concept.

its definitely an interesting problem. i think its possible that the elite journals are 10x better at detecting fraud but are 100x more like to be targeted. i wonder how we could collect data on this if at all possible

I think the problem is with the publishing culture in some disciplines. For an example of a field in which peer review does not appear to be problematic consider finance/econ. In finance and econ you march you paper around to a number of different conferences. At AFA you even get a discussant that presents a critique after you present your work. After evolving the paper based on feedback from the conference circuit you eventually submit to a journal. At this point your work has been (hopefully) widely circulated in the community and is unlikely to be some piece of crap.

Contrast this with engineering. When you submit to a conference it has to be "new" work and you lose the copyright. It is a one shot deal. So, the benefit of feedback you might get on your work is basically irrelevant because that work in no longer publishable.

I don't think that's always true however. Check out this example of an economist featured on Retraction Watch: http://retractionwatch.com/2015/01/05/anyone-want-hire-econo...

That said I do think the idea of a paper being produced after it has been toured around is a good idea - I wonder if it is appropriate for all fields though?

It can work for e.g. theoretical physics. But for anything involving serious extermination you've got to take the word of the researchers that they honestly did and reported it. At the most extreme example, experiments done with the CERN Large Hadron Collider are not easily replicated elsewhere at this moment ^_^.

It could also work for the explanation parts of experiment based papers, but, eh, you're allowed to be wrong there. Just publish enough of the data allowing others to double check your interpretation.

It is left as an exercise to the reader what other fields are known for withholding data, and the likely reasons why.

Reply on news.ycombinator.com