Comment by bmastenbrook | Hacker News Reader

Comment by bmastenbrook | Hacker News Reader

Parent Full thread

My experience being on the reviewer side of the process is very limited. As a frequent reader of papers in CS I'm of the opinion that peer review is accomplishing very little outside of checking theoretical results with proofs that can be vetted. I can count on one hand the number of papers I've read in the past month or two describing an implementation of a technique where the implementation described was available publicly. In all other cases, I had to take the authors' words for it. Given the poor handling of data for publication I've seen in the past, I'm not especially inclined to do that.

I also don't believe that the current conference / journal system for publication of results works especially well. There's a well-documented bias against the publication of negative results. It encourages authors to "save up" interesting work for publication when pieces of it could have been independently examined much sooner. In combination with "publish or perish" metrics, it discourages valuable or interesting work that might not ever rise to the level of a full journal publication. The format is incredibly stifling; important information is left out to satisfy page limits, code mangled to fit within columns, and interesting data visualizations not even considered because of the constraints of the media.

I think it's inevitable that the current common publication system will be recognized as a relic of the paper era, but I fear that any fix will only make superficial changes to the system. I think it'd be much more useful to have a publication system that allows work to be published with full data in a format that allows evolution in response to open reviews. This obviously wouldn't work well in areas where protection of patient privacy requires data obfuscation; I don't have a good answer for that. In all other areas, especially where the research is funded by public money, full data and code transparency ought to be key, especially where raw data must be adjusted or normalized in order to be useful and in situations where bugs in modeling programs could subtly affect results.

Edit: see also this excellent article from last month's Atlantic: http://www.theatlantic.com/magazine/print/2010/11/lies-damne...

"As a frequent reader of papers in CS I'm of the opinion that peer review is accomplishing very little outside of checking theoretical results with proofs that can be vetted."

As a frequent reader and reviewer of CS papers I totally disagree. Peer review helps narrow a huge incoming flood of new research into a smaller stream of results that are the most novel, important, and correct. Without explicit review, your options would be to either waste your time wading through the mess or use reviews from sources whose advice you consider on an ad hoc basis. I'm sure there are better systems that take advantage of the Internet, the wisdom of crowds, data aggregation and mining, etc. The current system of peer review does adequately perform a major service, though.

Given the huge number of niche conferences and journals out there, is the filtering that takes place actually useful? I'm not convinced of it. I've engaged in venue shopping myself. Meanwhile useful, novel work is published in the open every day and my brain hasn't collapsed from the inrush of poorly filtered information. I run across junk, but I find published junk too. At least the former usually has the decency to not waste my time with a bunch of academic boilerplate language and a page of barely relevant citations.

Niche venues cater to specialists in an area. Even in specialized areas there is more output than an expert can keep up with, and these smaller venues generally perform valuable filtering as well.

"my brain hasn't collapsed from the inrush of poorly filtered information" Of course not. However, you are either wasting a ton of time or you are using some other indirect method to decide what papers are useful to read. I'd love to hear if you think your personal approach to filtering non-reviewed content is superior to relying on peer review.

Once again, I'm not convinced the specialists are actually filtering usefully instead of juat defining the boundaries of their niche.

You've assumed that the non-reviewed content I'm reading is in the form of papers. It isn't; it's blog posts and code.

The internet by itself can play role of the peer review system. For example, github is a much better medium to publish a work that needs to be examined (reproduced) than any paper journal.

    I can count on one hand the number of papers I've read in the past month or two describing an implementation of a technique where the implementation described was available publicly.

so... less than 32 papers have the implementation available publicly?

Joking aside, how many papers have you read in the past month or two? I assume the number is at least in the double digits.

I honestly don't remember. It's somewhere between 25 and 50, though not all of those were read in detail.

In my experience, the biggest problem in CS academia is reproduction. It seems that very few people spend time attempting to reproduce results from academic papers, even though that's an important part of the method.

Reply on news.ycombinator.com