Comment by bhouston | Hacker News Reader

Comment by bhouston | Hacker News Reader

Parent Full thread

I've participated a bit in academic paper reviews over the years for some ACM journals/conferences in the computer graphics area. Initially I was pretty green and I often would not catch some of the problems that the more experienced reviewers would catch. I embarrassingly recommended acceptance to some papers that other more experienced reviews said were clearly crap. Over time though, I learned to be more critical by example from the more experienced reviewers. And eventually I sometimes would be one of the assholes on the review committee that wrecked people's dreams of publication.

I wonder if the rapid growth of ML recently has diluted the reviewer pool dramatically? There are so many papers submitted but so many of the reviewers are green that crap gets through more easily? I wonder if there is a growth limit to fields such that the paper review teams do not get overly diluted with green researchers?

(Has this paper even been peer-reviewed? If it hasn't been peer reviewed there is a good chance it is crap just by the law of averages -- most "academic" papers are crap. There is a reason the top venues that I was involved with have a rejection rate upwards of 80%.)

>I wonder if the rapid growth of ML recently has diluted the reviewer pool dramatically?

There was a blow up a couple months ago around this issue. Someone posted to r/machinelearning saying they were starting grad school in the fall, and had been accepted as a reviewer for NIPS2018.

Twitter convo about it [0] generated some press, with the pull quote of "It's 'peer review', not 'person who did 5 TensorFlow tutorials review'".

[0]https://twitter.com/yoavgo/status/996021724188725248

Some machine learning conferences are now recruiting graduate students to be reviewers.

This is a common practice in a number of other fields (e.g. cognitive science and psycholinguistics), at least for conference submissions. In general I don't see a huge difference -- when a grad student has sufficient standing and expertise to be chosen by a reviewer, they generally know the domain very well and are often up to date with the research in a way that more senior reviewers often aren't. And because they have fewer demands on their time and feel a greater need to substantiate their critiques, they tend to write depthier reviews. And of course there's still the safeguard that the meta-reviewer can indicate to the authors in various ways that a review is garbage, or even throw it out / seek another reviewer.

I think a lot of what you're saying is valid, but to be completely clear: these are graduate students who are reviewing for tier 1 conferences while being yet to publish in a tier 1 conference themselves. There is a legitimate argument that they simply don't have the academic maturity, experience or competency to be reviewing other researchers' submissions, regardless of how well intentioned they are or how much they want to prove themselves.

Have you read the paper or just the review? I'm interested in your opinion as an experienced reviewer whether this article appropriately evaluates the paper.

I knew a specific sub-field of computer graphics well enough to be a paper reviewer and even then I wasn't initially very good at it. I am not competent to review anything outside of that specific sub-field.

Reply on news.ycombinator.com