Is this brittleness stopping the scientists achieving what they need to achieve?
Are you sure that writing tests makes science better? Or are you just assuming that?
They aren't idiots and they aren't ignorant of how professional software developers work.
Academics have a lot of excuses for writing terrible code. It's really shocking. So far I've seen:
- It doesn't matter if the code is buggy because the results don't need to be accurate
- We can't afford to write good code.
- We aren't properly trained to write good code.
- We just average the results and that fixes the bugs (!)
- We aren't computer scientists so what do you expect.
- Science is special which means we don't need to write tests to get correct results. We just eyeball them and know if they're right (so why bother writing a model at all then?)
- You can't criticise or judge us because mere software developers can't understand science.
- Nobody told us it's easy to screw up memory management in C
And a whole lot of other baffling and insulting nonsense. How many of these excuses would be accepted if a private company produced a dangerous product due to bad code, and produced this litany of BS in a courtroom? None.
They aren't idiots and they aren't ignorant of how professional software developers work.
It's apparent from some of the responses to this fiasco that they are totally ignorant of how professional software developers work. And they're proud of it, which in my view makes them idiots too. You can't both tell governments and whole societies to "follow the science" and then blow off any suggestion of working to professional standards.
It is perfectly acceptable to write code without tests. Proof-of-Concept or Minimum Viable Product are a great place to write code without tests.
It is less acceptable if other people will run or use that code. It is even less acceptable if anyone (including oneself) ever updates or extends the code.
---
You could take this analogy to scientific instruments. Imagine you make a novel particle detector. You get a scientific result with your detector.
A colleague uses your detector, but they don't clean it properly before use, and they use a power supply with lower voltage. They don't detect any particles! Was your science bad? Would the "science be better" if there were clearer instructions and pre-requisites?
Now imagine another scientist makes a copy of your detector from the description in your paper. They get some stuff wrong because your description was ambiguous. Was the science bad?
---
By the way, all of these things are real problems with scientific investigations, and not just in the software realm.
And yes, this is a serious problem in science.
I don't understand why having brittle code would mean that the results are not reproducible? You don't need to modify the program to do a reproducibility study.
Science has this additional problem that its memory is short - mistakes seem to be discovered when work is in the long tail of the citation curve, once it's out of the news. Even if you retract a paper, there is no easy way to trace the contagion to the work that uses it. That's before you consider mistakes that might be deliberate [3].
I have no doubts that more software/data rigour would make science more accurate, but the cost would be substantial, and it would no doubt slow down discovery until the benefit of open source kicked in.
[1] : https://theconversation.com/the-reinhart-rogoff-error-or-how... [2] : https://www.the-scientist.com/news-opinion/mistaken-identiti... [3] : https://retractionwatch.com/2016/09/26/yes-power-pose-study-...