How Bad Software Leads to Bad Science
motherboard.vice.com
motherboard.vice.com
The study by the SSI, only proves that people are writing software, and that many of these people have no training in software engineering. The SSI claim this training is necessary to prevent things like the retractions of journal articles they discuss in the article. But their survey doesn't go as far as to provide evidence that there is a link between formal training in software engineering, and errors in scientific journals.
If this article was science, I would call it bad science, because the data they present is not sufficient evidence for their hypothesis.
Problems arise when that software is designed by researchers who really don’t know what they’re doing when it comes to coding. A single mistake in the code can lead to a result that appears innocuous enough, but is actually incorrect.
Oh yes I'm getting a bit sentimental now and it's not like that everywhere and not every researcher writes code like that. But if even me, with over 10 years of programming experience and all my automated builds and tests and whatnot cannot reliably produce code without one mistake, how on earth are researchers for which the programming part is often just a byproduct, a necessary evil, going to do that?
In my experience these sort of bad outcome occurs when a senior supervisor gives a student or junior post doc a project that involves programming, but is not able to provide adequate supervision. Instead of finding a collaborator that can help they just let the junior scientist thrash around on their own.
Especially in programming, where you want to actively avoid reinventing the wheel as much as is possible.
Hell, I've seen people come out of computer science PhD programs with poor software engineering skills and habits. If CS programs don't reliably teach these things, how can you claim researchers in other fields automatically have them?
Two other possibilities:
- Research code is uniquely difficult in ways that are different to other kinds of software. This is because the logic is often inherently complicated, since you are doing a complex calculation.
- Researchers are stuck using old fashioned languages and tools because they don't know better. Note that this doesn't necessarily they need training, only being pointed to the right places.
There is no valid reason why this should not already be a universal practice. If you are looking at any paper which cites the output of a custom-built computer program but omits the source code, it's probably safe to assume that the program is a mess and therefore the conclusions are not (yet) robust.
While it would be great and preferable to have both, isn't the code just a language/stack/programmer(s) specific interpretation of the underlying model?
There's the catch -- it should be, but there's no way to verify it's correct without comparing the source code against the model. (Well, I suppose if you had access to the raw data and the time to rewrite the software yourself from scratch, you could do that and see if you got the same results. But who has time for that?)
I'd definitely prefer to have both a model and an implementation available, but if I had to pick just one it would be the code.
This occurs because many of our people take the code cademy course on python for example and then assume they "know how to code" when they finish; if they run into problems, they can simply consult stack overflow for anything they don't know. They therefore never learn anything about formal software development best practices unless they put in extra effort to do so, which they generally don't.
I have learned a lot trying to refactor over the last few months, and I have tried to pass on some of this knowledge to my labmates. We have actually come a long way over the last few months. We now employ version control, use style guides, and more robustly document programs. I am currently trying to sell my labmates on the merits of unit testing but we'll see how that goes :)
Not to mention the publish-and-forget.
Also publish and forget... that's just the natural fallout of the way research is conducted and funded. I've written scientific software that I don't feel much personal obligation to maintain or update. It's challenging work which has greatly reduced payoff (to the powers that be) after it has been tallied on the scoreboard. Also, most positions in scientific labs are on a clock; not exactly an environment that fosters long-term stewardship.
I'd love to see science funding agencies spend a couple million to fund reasonably sized groups of 5-10 programmers/scientists to produce code over ~3 year funding cycles. Just to make sure that signals don't get crossed, these groups should be told not to spend a single minute of their time on publishing to journals. Dissemination is limited to short talks and training workshops. Continued funding would also focus more heavily on user surveys than third-party citations.
Now moving to a diagnostic setting, and working by myself, testing and documentation have been the main goals to me. And now, every single piece of third-party software has to be exhaustively validated in every aspect before being put into production mode.
"I'd love to see science funding agencies spend a couple million to fund reasonably sized groups of 5-10 programmers/scientists to produce code over ~3 year funding cycles. Just to make sure that signals don't get crossed, these groups should be told not to spend a single minute of their time on publishing to journals. Dissemination is limited to short talks and training workshops. Continued funding would also focus more heavily on user surveys than third-party citations."
On the side of software/funding, your ideas are spot on. I think it is about time to have professional and steady funding to keep software working and available. I haven't published app notes or the like in the past, most of the software I worked on were never published, but I see important applications where the code is not open source, there's no testing and no updates for years. We need to move from the on-the-clock science to some model that would allow scientific software to be properly maintained.
Maybe if we treated scientific software like we treat "regular IT", might be the key.
My main exposure to this has been by observing a biologist draw absolutely unsupportable conclusions from a bunch of very low quality data using some software as if it was a magic incantation. Not that that bothered supervisors or anybody else, as long as the funding kept coming everybody was absolutely happy.
The software wasn't all that bad, even though the user interface was crappy. The whole pipeline did not stand up to scrutiny, and of course my sample rate of this sort of thing is too low to draw conclusions from but I really hope that was an example of how it is normally not done.
Personally I think that if you aren't using software to do something faster that you could do yourself manually then you probably shouldn't be using that software.
Edit: you think this is hyperbole but I've heard the above two statements more than you might guess, coming out of the mouths of prominent scientists.