Hacker News Reader

Comment by karencarits | Hacker News Reader

This is also one of my fears, and my strategy has been openness - I share my analysis/code with colleagues/Github/as supplement. Maybe they can spot errors or reuse some of it, and I feel that there is some "protection" in it

View on HN

It should be a requirement that every paper be completely reproducible, for any software part of the paper. On any OS or systém, it should work. Similar to requiring statistics to be reported a certain way, there should be guarantees that the code should be available and work in all circumstances.

Unfortunately, I don't think this is realistic. Is it provable?

Does it require use of an "official" programming language and libraries, that are guaranteed to be maintained? Is Turbo Pascal for MS-DOS still supported? That's what I used for my dissertation, in 1993.

It's a far cry beyond what's expected in branches of science that are not mainly about programming.

Perhaps require making a docker container and writing a test script that includes inputs and expected outputs. There could be a website that loads those docker images and shows that the tests pass. Tests could be referenced within the paper. Would still require scrutiny to make sure that code doesn't hard-code the outputs.

This should be the way. Folks will forgive sloppy code on a “fast” paper. However it may invite more replication attempts.

Many folks have messed up simple algorithms before.

Reply on news.ycombinator.com