Hacker News Reader

Peer review is broken not because the process of review is broken but because the peers are broken. Reviewers, like all academics, have screwed up incentives to get there name on as many papers as possible and those papers cited widely, regardless of the quality of the science involved. They will always find a way game the system to their benefit.

Recently heard about reviewers of a friend's paper blocking publication until the reviewer could publish his research on the same topic first. The whole system is just too screwed up to be fixed.

Yes this sort of bad thing happens, but more in the real high stakes games of grant reviews than publications. You can easily kill the grant of one of your competitors by just giving it a mediocre review (without any basis of course). I would see this happen all the time to my colleagues where they would get a spiteful review on a grant that would kill it and there was nothing they could do.

At the NIH, at least, I know that enormous effort is put into ensuring that there are not conflicts of interest between the reviewers and those being reviewed.

I'm not particularly surprised to hear that some places do not put in that effort - it is a slow, painstaking process - but there are places where this is less of a problem.

I am in Australia where the pool of potential grant reviewers is pretty small for most areas. When you have a grant success rate of 15% it is all too easy to nobble your competitors via the review process.

Why is science viewed as a competition there?

For exactly the same reasons it is everywhere in the world: because there isn't enough funding to go around - and the grant application process is explicitly designed to be competitive.

It doesn't seem to be everywhere though. I know in the US, research is often collaborative and a single grant may fund many researchers. Those researchers do not seem to be competing internally for funding (though, I have seen conflict as to whose name should appear first on a publication!). I presume the same is likely true in Australia. Why is some research able to be collaborative while other must (apparently) be competitive?

You are quite mistaken, imho. The pay line for NIH grants is around 15% and for NSF grants it's more like 10%.

Note also that collaborations compete with other collaborations for program grants, and R01s (the measure of a "real" PI) are inherently not collaborative. They are by their very nature competitive. And they are the metric by which principal investigators are judged.

Many real PIs support collaborating researchers and students though.

Right, but students are written into the grant. Ultimately, nobody gives a shit which post doc or student you put on a project; the assumption (often flawed) is that they're all the same as far as the modular budget is concerned.

If they were special, the reasoning goes, they'd have their own F32 or T32 to work on the project.

Because there's a finite pool of money.

It is a competitive everywhere, just in Australia you are playing in a much smaller pool. There just is not enough money to fund all good grant applications.

I'm only shocked that this is the first time you'd heard of adversarial delaying. When that fails, sometimes PIs will call up their editorially inclined pals and suggest a "bundle" issue to sneak in their lab's work. There are many ways to avoid ceding priority for discovery.

If preprint deposition is the default, this all goes away. The people who lose from such a shift are editors of major journals, publishers of journals, and senior PIs who know how to rig the game. A cynical mind might propose that these are the major forces resisting such a shift.

> and even worse it rarely improves the quality of the paper being reviewed.

That might be your experience, but I think it has significantly improved the quality of a large fraction of the papers I have been an author or co-author on. I have encountered one bad reviewer in the past, and in retrospect perhaps I should have taken more notice of his report. As a reviewer, I've only had one negative incident, which was when my comments were overridden by the editor. My experience might be specific in my subject, as we only have one referee per paper.

I am curious as to what improvements the reviewers made to your papers? What areas did they improve?

In the case of the one paper of mine that was improved one of the reviewers noticed a connection between our data and a paper published 10 years before in a different field that made our work much more interesting. Of course after incorporating this the editor rejected the paper despite all three reviewers saying it should be published.

The editor of the next journal I sent the paper to was happy especially as the paper has since picked up over 1000 citations, but such is the vagaries of peer review.

The suggestions have included alternative explanations for the results, useful checks on the data analysis procedures, sensible ways to restructure a paper, useful plots which could be included, places where the text was unclear, indicating where our claims were too strong, how to strengthen the paper by comparisons with theory.

One referee's questions were useful to reveal that an undergraduate's paper (of which I was a co-author) had some fairly serious flaws which required it was withdrawn. Fortunately this has been the only "rejection" I've suffered so far, other than papers speculatively submitted to Nature or Science which were later published in mainstream journals. In astronomy and astrophysics we are fortunate to have only 3-4 major journals with similar impact factors.

I wish that was the way the biological sciences worked.

My personal favourite peer review process story was from one of my first papers where I wrote up the whole thing and my supervisor added his name and sent it off. This was in the pre-electronic days where you physically posted off the manuscript. Serveral months when by and we heard nothing. After around six months of silence my supervisor decides to ring the editor and find out what is happening (proactive he was not). The editor tells him "oh didn't you know that paper was published a couple of months ago" - not a single change made or reviewers comment to deal with. I wish all my subsequent papers had gone like that :)

I think that example is a bit unfair, teaching really isn't scientific when you get to the core of it. You cannot expect professors to be able to conduct double blind studies with student grades and so a more overarching non-scientific discussion is probably more amenable to actual improvement in performance.

You can actually conduct scientific studies on teaching and grading, but that is not the point I was making. What concerned me is how a department full of scientists wasted their time discussing the importance of results which were statistically meaningless. The scatter was just noise yet we wasted hours (days) of our time discussing the supposed importance of this noise.