Hacker News Reader

Comment by dr_kiszonka | Hacker News Reader

One of my greatest professional worries is publishing a paper with an honest data analysis error and being accused of fabrication. For this reason, I spend an "unreasonable" amount of time triple-checking everything (my colleagues are annoyed by that).

I wish there was software that would help me spot mistakes by looking at my code and/or write up, but I also recognize that it would be abused by those committing fraud.

View on HN

This is also one of my fears, and my strategy has been openness - I share my analysis/code with colleagues/Github/as supplement. Maybe they can spot errors or reuse some of it, and I feel that there is some "protection" in it

It should be a requirement that every paper be completely reproducible, for any software part of the paper. On any OS or systém, it should work. Similar to requiring statistics to be reported a certain way, there should be guarantees that the code should be available and work in all circumstances.

Unfortunately, I don't think this is realistic. Is it provable?

Does it require use of an "official" programming language and libraries, that are guaranteed to be maintained? Is Turbo Pascal for MS-DOS still supported? That's what I used for my dissertation, in 1993.

It's a far cry beyond what's expected in branches of science that are not mainly about programming.

Perhaps require making a docker container and writing a test script that includes inputs and expected outputs. There could be a website that loads those docker images and shows that the tests pass. Tests could be referenced within the paper. Would still require scrutiny to make sure that code doesn't hard-code the outputs.

This should be the way. Folks will forgive sloppy code on a “fast” paper. However it may invite more replication attempts.

Many folks have messed up simple algorithms before.

I do the same. I write and prepare figures for each single paper that comes out from my lab. I double check each single Jupyter notebook and I share every single piece of raw data ( for the last paper we just submitted yesterday we uploaded 316Gb of data on zenodo). All this takes me weeks of course. I would be very skeptical of colleagues who publish more than 4-5 papers a year.

Are there (m)any examples of that? Especially given the fact that exposures of clear fraud when exposed is so often just ignored

Reply on news.ycombinator.com