Hacker News Reader

Comment by lacker | Hacker News Reader

I was shocked in CS graduate school the first time I downloaded the code referred to by a very well-respected paper in the field, and found that it did not even compile. By the 20th time, I was less shocked, but I had lost a lot of respect for CS academia.

The correct standard should be, if you claim you wrote a program that does X, the program should be available along with simple instructions that explain how to run it. The reviewers should then follow the instructions and verify that it actually works, and returns the results described in the paper! This basic process would sadly invalidate most computer science research.

View on HN

The vast majority of such papers don't really involve a claim that they wrote a program that does X. I mean, they probably did write such a program, but that's tangential to the paper and not particularly relevant; the fact that they did isn't one of claims in the paper, it's not something they're trying to prove/show in the conclusions. There are some papers in the style "Software package for doing X", but those are a minority and generally limited to major releases of widely used open source tools for that problem; most CS papers are not about software, just like most astronomy papers are not about telescopes.

E.g. the paper is about some new method to approach X. In general, the paper could be valid even without any program implementation whatsoever; but you'll likely might supplement the method description with some experimental evaluation - but the paper is not about the experimental evaluation or the "experimental apparatus" i.e. the code they used. I mean, the paper is likely making some claims about the method as such, not about any single particular implementation of the method including their own. As a part of the main claim "this method seems good and interesting" they're providing some evidence "we tried, and in certain conditions described here applying method X was 10% better than method Y" - but the actual code used to do that is just supplementary material; the code is a tool they used in research, not the result of the research they published. The code is the "telescope" you used to make an observation, but the paper is about the observation, not about the telescope.

It's just as with medicine - we generally accept papers that say "well, we tried this procedure on 100 patients and 73 of them got better" without requiring video evidence of those 73 patients actually getting better; in the same manner, we accept CS papers that say "well, we tried this procedure on 100 datapoints and 73 of them got correct results" and don't require the reviewers to reproduce the experiments and verify if the experimental results aren't falsified; just as reviewers don't try to reproduce experiments in pretty much any other science.

The difference is that when a medical paper claims

> well, we tried this procedure on 100 patients

"this procedure" is the most important part, and they describe it in detail in the paper, hopefully well enough that someone could attempt to replicate it, and some do attempt to replicate it. (Not that procedure descriptions in such papers are always sufficient for this.)

The difference between that and

> we tried this procedure on 100 datapoints

Is that it's nigh impossible to describe a ML procedure in enough detail to reproduce with just the description in the paper. Tiny changes in the parameters and construction can completely change the result; the only way to be able to reproduce it is if you had the source code. And also the source data which is just if not more important as the source code (see sibling thread).

The opportunity that academic CS has over every other science is that they could empower every reader with the capability to verify the results of every paper they read, and this is actually attainable. Reviewers of other sciences don't reproduce findings themselves for purely practical reasons that don't need to exist in CS.

To try out a procedure on one dataset amounts to a data point. You need to process a bunch of datasets to establish the performance of the procedure under study.

I though ML folks would have the statistical background to know you cannot infer a true statement from a single occurrence?

Hmm, it probably depends on your subfield of computer science. These AI papers almost always claim they wrote a program that does X, and so did the stuff I was working on in grad school, but more theoretical things probably are more similar to what you describe.

You're completely wrong. So are most people in the comment section.

Research papers should NOT release code. Ever.

Researchers don't care, and should not care about your setup, your IDE, your OS, your dependencies, your processor architecture etc. None of that matters, unless it's the central point of the paper. You literally ask researchers to write coding tutorials or something. It's simply comedic.

Most machine learning and CS research revolves around Methods, Architectures, Processes and other abstract approaches to solving a problem. Their algorithms are always reproducible with any implementation you like.

Sure, the training data should be supplied, because in AI/learning it's part of the algorithm, but even if sometimes they aren't, you can try and reproduce the relative changes with another similar database. If their point is that one training method works better than another, the result still should be reproducible with similar datasets.

I get where you're coming from, after all most people on this site are programmers, but research is not about publishing code that works, it's about publishing ideas that work.

The big issue is the low-quality of machine learning / applied CS literature in general. Pull a paper and try implementing their algorithm(s). You'll be almost certainly guaranteed to encounter barriers ranging from mistakes in the paper, omissions / ambiguities in the paper, to even downright fabrications.

The entire field has a reproducibility problem, which implies that as a academic community we're just churning out pubs and not actually doing science.

I'm helping form Nature's foray into ML (https://www.nature.com/natmachintell/), and we're strongly considering enforcing a requirement that FOSS source code be published in a public venue (like github), and requiring that at least some benchmarks are established on publicly (but perhaps not freely) available data sets. This won't be appropriate for every paper of course, but our editorial staff is certainly going to be focused on maintaining scientific quality.

Their algorithms are always reproducible with any implementation you like.

Well, that's the dream. In practice their algorithms are often not reproducible at all. Requiring code would get rid of the papers that cannot be made to work, where the authors stretched the truth because without code they could not be caught.

Reply on news.ycombinator.com