...And require the authors to publish their code, ideally in a container that runs out-of-the-box on the cloud.
2. the best research tends to outlive its initial implementation.
3. Some research code truly isn’t fit for publication, or is useless on its own. Eg anything that processes either proprietary or phi/pii data.
Number 3 aside, I agree there’s at least no reason not to throw code in docker and provide a basic README, but that practice is neither necessary nor sufficient for reproducibility. Science isn’t software development; the goals are different.
I say this as a researcher with tons of repos that all have unit tests, are super well documented, and have thousands of end users. In other words, I am a researcher whose career would benefit from greater emphasis on implementation quality. That’s one of the reasons I oppose the engineer’s perspective on “reproducibility” — implementation quality is important in science, but for different reasons than in engineering.
Confusing science for engineering puts the emphasis in the wrong place. The very last thing cs needs is for docker rituals to replace thoughtful critique of the implementation. I’ve seen lots of manicured code that doesn’t match the theory and breaks on even tiny changes to the input. In science, I care a lot more about those things than bugs, bad variable names, out dated dependencies, etc