Hacker News Reader

Comment by coliveira | Hacker News Reader

An important point in this article is the vital distinction between academic software and "general purpose" software. The goal of research software is to prove or exemplify a point explained in one or more papers. It should do so in the most direct and economical way for the researcher(s). It makes no sense to create multi-platform, software-engineering friendly software for this kind of use. In the last few years we have seen an untold number of complains that research software is not robust, user-friendly, etc., and it entirely misses this important fact.

View on HN

The problem is that the "direct and economical way" tends to basically enforce bit rot and lack a reproducibility. It also tends to screw whoever inherits the project.

Reproducibility is an issue in all scientific fields, it is not something unique to computer science. Can someone easily reproduce experiments performed in a particle accelerator? What about an extremely complex chemical reaction? The answer to these problems is to have more people doing research in that area to make sure that the problem is well understood, instead of requiring that scientists slow down the research because it needs to first achieve some kind of "engineering reproduction" standard.

Speaking for the chemical reaction - yes. That is exactly what should be possible. Some reactions are hard to reproduce and labs spend years trying to figure out how to do so. Without that you really don't have good science.

The key is .. can someone read your paper and reproduce your experiment and the results. If not, it's not valid; it's just a paper.

That's not true. Some types of chemistry cannot (legally) be reproduced, but they are still good and valid science. For a trivial example, consider the many chemistry papers based on measuring fallout effects from nuclear testing.

More prosaically, some things can be verified even when a chemical process cannot be reproduced. Protein crystallization is a tricky process with many irreducible results. Yet if you have a crystal, and can use the crystal to determine the protein structure, then you can verify that the crystal indeed contains that protein - even if you are unable to reproduce the chemical process used to create the crystal in the first place. Others can use the same sample to re-verify the resulting x-ray structure even though they also might be unable to reproduce the crystallization step.

My critique isn't aimed at computer science research--it applies equally well to other fields using computation to produce results.

There are a lot of projects (and I won't name names, because I've got friends here who work on them) whose results cannot be reproduced, because they ran on some weird-ass combination of a weird libc, using a weird build system, running on weird hardware environments (especially common in HPC), with weird one-off data sets.

They report the results using this setup, and then the setup is gone forever. The code is probably not kept correctly in source control, and generally the community just takes them at their word that the results they get are authentic.

When you run millions of iterations on some crazy quantum solver, it's hard to spot if you're wrong. If others can't at least run the same result, science becomes a matter of anecdote.

As hard to believe as it is for software engineers, science is not in the code, it is in the thought process that leads to the code and in the interpretation of the results. The code for an experiment can be lost forever, but if you did your job correctly you have described why, how, and where the experiment works. Future scientists can now "build on the shoulders" of this knowledge and use it to go further and validate the previous result or discredit if it is indeed incorrect.

People complain that published papers are full of incorrect results, but they don't understand that this is just part of the process. Incorrect results will be forgotten because they will never be validated. Correct results will be validated and used (cited) by new scientists as they try discover even more complicated things.

Things would work so much better if the software was written in a better way.

Though you do have a point, I think it's really not binary like that; you've given end-members.

There are a good number of scientists (myself included) who spend at least part of the time making applications for other researchers; it's best when the products are multi-platform, because there are a lot of scientists who only use one platform. These application can be anything from plotting and basic analysis tools to simulators of various sorts. They're more specialized than Evernote or something but are still designed to be installed and used by members of the community, because it's a waste of time for everyone to write their own finite element models (even if everyone knew how).

When done right, not only are these tools enabling but they become very widely cited. For instance, in my field (geoscience) a tool called GMT (Generic Mapping Tools) has over 6000 citations, even though I'd estimate 70% of papers that have used it in some form don't cite it. This is an enormous number of citations in the geosciences; most very famous and influential papers have ~1-2000.

Maybe it's different in CS; I have heard that much of the software is illustrating a new algorithm or something.

That seems rather short sighted. Science is often described as standing on the shoulders of giants, you can't do that if everyone has to start from scratch. Sure it doesn't have to follow NASA's coding guidelines but it also shouldn't be utter garbage that no one besides the authors can figure out.

Reply on news.ycombinator.com