It makes pulling out the actual content of the papers such a slog.
[0] exhibit A: https://ieeexplore.ieee.org/document/1017616
Source: am former producer/publisher of abomination research code.
Edit: Perhaps one way to encourage this is via topic "reuse" papers, which present further detail on a previously published papers topic, but this time with code and more detailed discussion. It would at least help the situations in that the author gets some reuse of effort, but in a way that still shows new information and advances the field.
distill.pub has a journal that strongly encourages interactive articles: https://distill.pub/journal/
It's a lot of work, tho:
> One good test for whether your article is a fit for Distill is whether your collaborators and you are willing to put in whatever time is necessary to write and illustrate an outstanding article. In our experience, this often takes 100+ hours.
Literate programming suffers from this in general because you don't want to clutter up your document with noise about versions and so forth - even as appendices, including package information and build instructions is a lot of noise.
Requiring that papers making claims about some code base publish that code base simultaneously with the paper via the peer review process should be sufficient to improve the overall state of affairs.
Don't underestimate the value of demos and competitions with companion papers either, those tend to get a lot more notice.
In general, you'll have a lot of approaches that don't work out which are nice to have a record of, but definitely don't merit being in the final paper.
If your claims don't depend on specific behaviours of some body of code, you wouldn't need it - eg, an article claiming some asymptotic performance of an algorithm should describe the algorithm in the abstract s.t. the performance bound can be proven, not some specific language's implementation of the algorithm.
But for most journals, it's the story as usualy. Personally I dont feel very confident about a paper result if they don't publish the code. Why hold back on the code?
Months of tweaking code is exactly what you want some test coverage for - so that you end up with a tweaked version of something where you still know what it does instead of something that does god knows what.
http://matt.might.net/articles/crapl
> The CRAPL is an open source "license" for academics that encourages code-sharing, regardless of how much how much Red Bull and coffee went into its production.
I know you know this, but this is absolutely a feature.
The big question is why this failure to publish code is tolerated by program+steering committees.
Besides, _ideally_, machine learning reseach is not about building a system but about making a theoretical claim that is supported by empirical results. The main claim should be about a property of the algorithm: its time complexity, its convergence, its correctness, etc. Experiments should directly test the theoretical claim and verify it in practice.
... ideally. In practice most machine learning papers are vehicles for a little table that compares your system with a baseline that you chose based on the fact that your system beats it (or you wouldn't have put it in the paper). Whether your experiments prove anything beyond the fact that you beat some benchmark is not really important.
And so whether someone publishes a reproducible paper is of no consequence because the only point of papers is to pass review and convince someone that the researchers should stay in the (research) money. In any case, things move so fast that most of the time nobody wants to reproduce your old and unsexy paper from last week.
Then some busybody comes along and points out the fact that you're wearing no clothes. Like in the paper above.
Also, more likely and non- nefarious, but no less harmful, there's just an academic culture that says "this is what a paper looks like and this is what it contains". Moving from this pattern increases the odds you'll not be published, and it would also likely result in a scandal when someone uses it to point out that the peers in peer review are neither peers (they can't read nor understand nor test the code) nor reviewing (they won't actually put resources into code review).
As a non-wealthy, non-academic, this isn't a battle I'm interested in fighting directly. There are other hills I'd rather die on :)
In some cases I basically had to run a test harness and brute force the parameters they'd used, because the algorithm was basically useless without picking parameters in the right range.
It was hard to accept then (~2008-ish), but now we're at a point where frankly at least for the more basic CS papers we ought to be able to expect source and some runnable format (e.g. a VM image) with test data.
I handed in a DVD with gigabytes of test data, full source, built binaries, and a test harness for the work I did for my thesis. All it took was to follow practices that I needed to follow to satisfy myself that my results were valid anyway.
I realize that for some work that relies on much larger and/or proprietary datasets making everything available would be hard, but a small test case illustrating the code run with the right parameters etc. ought to be a minimum unless there are substantial barriers, in which case the paper ought to signpost clearly what is missing and why.
The paragraphs of English text help them to be able to claim that they are indeed the authors of some novel method (exactly what the author wants).
Code/data accessibility helps others to both find errors/imperfections in their approach and to build upon their results (exactly what the author doesn't want).