Comment by dash2 | Hacker News Reader

Comment by dash2 | Hacker News Reader

Parent Full thread

My experience is different. Peer review is far from perfect but:

* Most of my papers have had thoughtful comments. Typically, they improved the paper, sometimes a lot.

* I have very rarely been asked to quote a specific paper from a specific author. Of course, pointing out important references that the paper has missed is part of peer review, so this would not necessarily be evidence of corruption.

* There are loads of crap pay-to-publish predatory journals, but publishing in those journals would harm my career, not help it. Nobody cares about them.

* I've not experienced people rejecting papers from rival labs or departments. I don't say it doesn't happen, but I haven't experienced it.

* I myself try hard to give high-quality reviews that explore the paper's value in depth.

I am from one particular discipline, country and subfield. Others may have very different experiences.

Again, I am not claiming academic peer review and publishing doesn't need a lot of improvement. Removing the parasitic mainstream publishers like Elsevier and Springer would be a great start.

It varies field-by-field. In my field, peer review is a joke. I'm glad it's not a joke in your field yet, but incentive structures say it eventually will be. If incentives don't change, it's a question of when, not if.

What incentive structures? When I get a paper for review, I do my (time limited) best. Which incentive will change me?

Academic hiring and promotion.

High-quality academic curation doesn't effect anything in your career. On the other hand, the impact of your own research does does affect it. I'd like my taxpayer dollars to go a little less towards publish-or-perish, and a little more towards QA to make sure the papers coming out are credible.

I agree with the general point of your comment, but my personal experience with the usefulness of reviews is not so great.

Nearly all reviews were superficial, not a single review ever found any technical flaw, even when some were discovered later. The acceptance recommendations seem to be governed by the overall impression the reviewer had from the paper and not by any factual points they make in the review. This manifested in getting 5 reviews at one conference, where all reviewers made nearly the same points in their review: all about the presentation of the paper. The paper received all of the five recommendations: Accept, weak accept, borderline, weak reject, reject. (The most critical and useful with a small correction of one formula was the "Accept" review.)

Many reviewers also leave the impression they didn't read the paper in full or at all, or ran out of time while for the review, so looked for a reason to reject the paper and didn't look any further.

Note: I am not saying my papers should have been accepted, 2 had serious issues I glad I discovered later! I also had a few very useful reviews to improve the presentation of my paper.

I largely agree with this comment, but this just means the underlying issues are more subtle. For example:

* I typically get thoughtful reviews, and in particular critical reviews are often thoughtful. Still, I know that some people assert influence on the outcome of the reviewing process of their own papers. Also, there's a fine line: for example, many journals allow you to recommend reviewers for your papers. If you're well-connected, you can recommend your best friends and your editor friend follows your advice. Is this misconduct? I'd say it depends. After some decades in a math-heavy community, most people who are experienced and competent in the context of a highly specialized sub-field will be your "friends".

* Journals that are outright predatory are, of course, a no-go. Still, there are edge cases like some MDPI journals; sometimes there are special issues from credible, relatively junior people in my community. Should we support them or not? Also, my funding provider wants me to publish in high-profile, "big publisher" journals and pays the open access fee; these journals typically rank much higher than community-owned/not-for-profit venues. On the long run, it's better to fully move to community-owned venues only, but as a junior researcher I feel I cannot afford it at the moment.

Personally, for me the problem does not come from outright and obvious ethics violations; sure, I know of some individuals who I consider somewhat problematic, but I don't think these people control the community. For me the problem is to navigate all the grey areas, knowing that the system is largely ethics-agnostic.

I find it intriguing that reproducing the experiment isn't a part of the review, or visiting the lab to look for potential flaws in their set up.

These reviews seem far from what you hear about for disproving N rays and the like

Do you checkout your co-worker's branches and run the unit tests for every PR they make? In an ideal world that might be nice but in the real world with normal sized projects that's to time consuming so everyone just reasons through patches.

If reproducing were a requirement for reviewing, publication cost might be exorbitant (like when CERN tries to publish a paper you have to rebuild the particle accelerator?)

So how do you explain the information in the OP?

I don't know enough about the specific situation of computer science at the University of Florida to add anything useful. It sounds like a serious scandal.

yes, this is localized and the system is not corrupt as a whole.

Reply on news.ycombinator.com