Reboot the Computing-Research Publication Systems | Hacker News Reader

Reboot the Computing-Research Publication Systems

m-cacm.acm.org

20 pointsazhenley4 comments

the difficult isn't designing such a system. the problem is getting people to use it, and having institutions accept the publications as part of a researchers CV. (come to think of it..if I'm a department chair - maybe I should be reading the papers of my faculty instead just trusting SIG* to do a good job).

I think a reputation system (ala HN, but a little more transitive) would work just fine here. and then the reviewers get some measurable benefit for doing a good job.

and then each paper thread just stays open persistently and people can ask questions, and reviews can get posted, and talks and slides attached.

then you can do inline comments and revision control

and then people can collectively manage the topic taxonomy and publish things like field survey papers.

...And require the authors to publish their code, ideally in a container that runs out-of-the-box on the cloud.

1. That’s not sufficient for reproduction. Reproduction would be something closer to reimplementing the code from the description in the paper, perhaps using a slightly different dataset as well. not simply docker running something.

2. the best research tends to outlive its initial implementation.

3. Some research code truly isn’t fit for publication, or is useless on its own. Eg anything that processes either proprietary or phi/pii data.

Number 3 aside, I agree there’s at least no reason not to throw code in docker and provide a basic README, but that practice is neither necessary nor sufficient for reproducibility. Science isn’t software development; the goals are different.

I say this as a researcher with tons of repos that all have unit tests, are super well documented, and have thousands of end users. In other words, I am a researcher whose career would benefit from greater emphasis on implementation quality. That’s one of the reasons I oppose the engineer’s perspective on “reproducibility” — implementation quality is important in science, but for different reasons than in engineering.

Confusing science for engineering puts the emphasis in the wrong place. The very last thing cs needs is for docker rituals to replace thoughtful critique of the implementation. I’ve seen lots of manicured code that doesn’t match the theory and breaks on even tiny changes to the input. In science, I care a lot more about those things than bugs, bad variable names, out dated dependencies, etc

Research should be reproducible but some research would be too constrained by the cloud.

Reply on news.ycombinator.com