Why we can ignore reviews of scientific code by commercial software developers
philbull.wordpress.com
philbull.wordpress.com
1. The developers had a couple of bugs that cause non-deterministic outputs in weird settings. It's a randomized model so this isn't a disaster, but obviously it would be better to have all non-determinism derived from a single seed. However, none of the bugs point to scientific issues. Still, these are real bugs that should be fixed. And indeed: another team pointed out some issues, they triaged the report, and it appears that they've been fixing these issues.
2. The code doesn't run well multi-threaded, and so the scientists run it on a single core. This seems like a feature the developers have de-prioritized. The critics make a HUGE deal about how using a single core is inefficient, but at the end of the day: if scientists can get simulations done in a reasonable time using a single core, this seems like a valid prioritization of limited development effort. The critics know better, obviously.
3. There aren't a bunch of unit tests. But as the authors point out code is in development and being altered continuously to reflect new developments in society and new knowledge about the virus, so it seems hard to develop unit tests that won't become broken as the underlying components change.
4. The critics seem to have a strong grounding in epidemiology. I'm just kidding, they claim to be pure software engineers, but still insist on spending a significant fraction of their critique disputing the calculation of R0 and lecturing the epidemiologists on how they think it should be calculated. This is a giant red flag given that the critics' sole stated expertise is software engineering.
5. The developers are continuing to add features to simulate new aspects of the ongoing pandemic (e.g., new code to simulate the impact of contact tracing apps) instead of fixing every minor issue in the code. Given that we're in the midst of a deadly epidemic that is putting human beings into the ground, this seems like a completely unavoidable resource allocation decision to me.
6. The critics can't even be bothered to pretend their criticism is objective. E.g., the piece concludes with "on a personal level, I’d go further and suggest that all academic epidemiology be defunded." Yeah, sure, you're a totally disinterested software expert here.
I guarantee you if you look at any piece of software -- commercial or scientific, no matter the budget -- with the politically-motivated goal of tearing apart the author, you will succeed. I can say this having spent years doing security and expert witness code reviews for some of the biggest and smallest companies in the software industry. This piece isn't a critique, it's just a hatchet job. And not even a particularly creative one.
Any arguments that the developers of this code are somehow above needing to mechanically test, or formally verify, or independently peer review the code to ensure quality are trivially debunked by the history of bug fixes and changes in behaviour that we do know about, even before making any judgements based on the apparent attempts to hide that history and to conceal the original code from before they brought in programming experts to help tidy it up.
If you have indeed been an expert witness doing code reviews, are you seriously contending that you would not annihilate any case made in a court that hinged on the results of code like this?
This is much more difficult than reading a Git issue tracker, so obviously the critics have nothing to offer us here.
The rebuttal makes a couple of statements about how scientists spend their time to verify their work:
> Scientific software tends to be validated in a more ad hoc way. This is a domain-specific norm, not a matter of competence.
> Practically every scientist I know spends most of their time checking their results rather than coding, and I sincerely doubt that the Imperial group is any different.
I wonder if the team could iterate more quickly if some of these ad hoc validations were automated.
But it's not just a science question. That is the point of the criticism.
Assuming you are the Matthew D Green known for his cryptography work, surely you of all people understand that a poor implementation can have profound consequences even if the original mathematics is sound?
So what do you do with your limited resources? Do you tell the government you can't get them any preliminary data because your multithreading system doesn't work, and you need four months to refactor the code (even though you could just run single-core)? Do you tell them that because you have to write a complete unit testing framework, you can't give them any modeling data or update your model parameters, sorry, good luck with everything.
Do you tell them that another 30K people may have to die based on wrong policy choices because you have decided to prioritize "shipping-production-quality" software standards, even though you are not shipping your software?
You can pretend that this is not a trolley problem. But in this case it literally is. You are standing at the switch in the tracks, and on one side is a detailed set of unit tests and multi-core support, and on the other hand might be tens of thousands of living people. And you know your code better than anyone else in the world. What do you choose?
I don't have an answer. I'm saying the answer is hard, and we're not equipped to lay claims on it based on glancing at a Git repo.
Also, just to be clear, all of the specific issues with the Imperial code that you've mentioned here and in other parts of our discussion today, such as whether to rewrite for better multi-core behaviour or just use single-core, are coming from you not me. My argument is a much simpler one: when it started to become clear that this virus was a significant risk, the government should have provided the necessary resources and the expert advisors should have switched to a model where scientists did the science and programmers did the programming. I don't think anyone can credibly claim that there wasn't enough awareness that this could potentially become a serious issue and so that bringing in broader expertise was appropriate by the time we had things like COBRA meetings being held and secret expert advisory groups being assembled.
It started being developed in 2007. It's by now over a decade old, with a lot of work done in 2014. This program wasn't written quickly, it was re-parameterised with a few values for COVID and otherwise thinks it's simulating the flu. Some other pieces of code date back multiple decades.
You also appear to think that "limited resources" is a valid excuse. The UK science budget is over 8 billion pounds. Universities there churn out a large number of papers. The funding is there if they wanted to do less work but of higher quality, yet they don't.
And finally, just to make that clear: yes, saying "We don't have a reliable model for you" is an entirely valid thing to say to a government. You seem to think even low quality simulations are better than none. Humanity has been fighting disease for thousands of years without bug-ridden 15,000 LOC C programs to "help" us. Not having an academic model is a completely reasonable position to be in.
In particular there's a lot of evidence this particular model has already killed a lot of people: it predicted a surge in hospital needs that never happened. In anticipation of this surge a lot of elderly patients that had tested positive for COVID were discharged back into care homes, because the hospitals thought they'd need the beds within days. Those elderly people then infected the others in the care homes, causing the death rate to go up. If there had been no model, there'd have been no panic-induced mass discharge, patients would have been kept in wards until they were fully recovered and the virus wouldn't have been flushed straight into the populations of the most vulnerable. Outcomes could have been much better as a result.
> Do you tell them that another 30K people may have to die based on wrong policy choices "How many more people should die before...." - where have we heard this, ooh just a couple thousand times before?
Here's a simple fact, if you want to go this route: every extra 1% of unemployment rate (for US) leads to extra ~40,000 death a year due to a whole stew of reasons.
So, which one is better: have a team of professional developers use best practices and existing tools (like yes - testing framework) and minimize what seems to be a result of amateur programming done over the span of 15 years, or release an unreliable yet sensationalist report and cause tens of millions of people loose their jobs and livelihood?
@matthewdgreen , with all due respect you're avoiding answering to concrete questions and keep defending lousy programming on the ground of urgency and appeal to emotions. This is a mere demagoguery.
Perhaps, you can also share with us the _original_ code of the model, based on which the "Report 9" has been made? So far, as I know, those calls went answered.
All the criticism would be moot if the model delivered sensible predictions.
* You'll obviously run into the "testing on your training data" problem, but again this isn't much of a criticism. We're in the midst of an epidemic and scientists are rapidly adjusting a model to reflect developing knowledge. The alternative is to have no models.
> It's a randomized model so this isn't a disaster
What you're saying here is that no matter what distribution properties the "randomness" has it's ok to substitute one with another, right?
> There aren't a bunch of unit tests
It isn't a Microsoft Windows, you know, where a lack of unit tests and uncaught bugs could be potentially fixed in the next release (or not). Report 9 is out, the harm is done.
#4 isn't an argument as we aren't arguing about epidemiological characteristics of the code, but about its validity and correctness.
I am loosing the interest in this discussion very quickly, because it is full of straw-man constructions and intellectual dishonesty, starting with the fact that the original code/model has never been shown to anyone besides of a very small group of hand-picked developers to groom it into some presentable form. Pity...
This is hardly an unexpected conclusion given articles like this one and comments like yours, in which people defend buggy code on the grounds that academics shouldn't even be expected to produce work that isn't buggy. Academics are claiming they structurally can't produce correct, well documented code, not even when it's being used to advise governments.
Hence the conclusion that maybe academics shouldn't be writing that sort of code: how is that not an obvious inference to draw? You seem to believe that any criticism of academia is "politically motivated". It's the same attitude that can be seen in Imperial College's non-answer to the problems: you can't state the private sector has higher standards than academia despite overwhelming evidence it does, because then you're partisan and should be automatically ignored?
You're also an academic, aren't you? Surely that means you're the politically biased one here. How could it be otherwise?
The suggestion was not that academics should not write this kind of code. The suggestions was Not do not write bad code but all epidemiology be defunded. So who will do epidimioly studies and develop models of deadly diseases? The software engineers? ;-)
1. Or do they? Or may be they do in certain cases? How can you guarantee that if the output matched your biased assumption but the output is non-deterministic that it's actually proves the theory?
2. That's may be an excuse to some degree
3. Bad excuse, it's all varies by design of the app. You can write either easy to test and modify code or spaghetti-code. Even if the code very volatile investing in proper design of the app will give you a plenty of room for unit testing
4. May be there are some of them who actually do have a grounding. In the age of internet it's quite easy to learn the basics having some math background. It won't make you an expert in that field but will give you enough to spot obvious mistakes.
5. If they personally can't do that why not to gather community together and combine the effort? Especially when it's a deadly epidemic.
6. Or may be somebody takes the critics too personal. Take it easy, you can always void it by a phrase - "please send a Pull Request". But you need to be ready to collaborate and actually accept PRs.
The whole situation looks very dirty and it's hard to filter out personal attacs and false critics. Although there are definitely some valid ones. edit: formatting
The results from using this software have directly influenced political decisions that have resulted in the greatest mass curtailment of personal liberty in modern history, not to mention the rest of the response to a public health situation that is going to cost hundreds of thousands of lives around the world.
The idea that code that demonstrably contains bugs and exhibits non-deterministic behaviour, written in a style that has proven time and again to be resistant to mechanical testing, formal verification or expert peer review, should be defended because while it's written in such a style that doesn't matter because this code was written by scientists is... I don't even have strong enough words. It's laughable. It's patently absurd.
Sorry for the slightly ranty comment, but I am genuinely furious that people are trying to defend this work in this way. It is insulting to me as a professional software developer who has also worked on many scientific and engineering programs over the years. It is the antithesis of good scientific practice, where methods are transparently disclosed and peer review and attempts to reproduce (or fail to reproduce) important results are actively encouraged. And it is downright offensive to me as someone who can't look after close family members right now because laws made in no small part because of this software prevent it.
^^^Swallow your politics if you can for a moment and read. At a minimum it seems that the rebuttal here was being argued in very bad faith. Given the stage set, I was expecting bikeshedding about the Law if Demeter or JSON vs other format, etc...
Following good practices have nothing to do whether it is "commercial software" or "front-end software", as the author claims.
> While these checks could also be handled by unit tests, most scientists generally just end up with their own weird set of ad hoc test outputs and print statements. It’s ugly, and not infallible, but it tends to work well given the intensive nature of our result-testing behaviour and community cross-checking.
The sense in which this statement is plainly wrong is actually addressed in the original article:
> Regressions like that are common when working on a complex piece of software, which is why industrial software-engineering teams write automated regression tests.
This is the whole point. You can only run your "ad hoc" print-statement-based tests once off, which is why such "tests" are useless for finding regressions.
It is therefore rather irresponsible of the scientific community to allow such a bad state of code technical debt to live on for so long, when it’s dealing with a field that is inherently tied to the study of life-or-death scenarios (like we are facing right now).
The only way to avoid this inevitable conclusion of irresponsibility would be if the scientists behind the model warned that it cannot be trusted at all due to the potential of a large number of unknown bugs (did they?). Otherwise, either the scientists are irresponsibility incapable of understanding how to write reliable complex code, or were not honest about the trustworthiness of their code.
If there is ever an asteroid heading towards the earth, I swear that some segment of HN will write angry articles about how the hastily built nuclear-powered spaceship we sent to intercept it doesn't have proper unit tests. They'll demand that we delay the launch until 300 days after impact.
We knew this was coming for months. The code isn't that large, it's just horribly written. If the scientists had worked in collaboration with an expert team of technical programmers, they could have handed off their mathematical models and had a well implemented, extensively verified program to run those models within a useful timeframe.
That might sound extreme, but once again, I will reiterate that expert advice apparently based in part on the output from this program is being used to inform decisions that will affect whether thousands live or die and that are severely curtailing the normal lives of the entire population with other adverse consequences that nobody fully understands yet. Powerful responses require powerful justifications.
If you look at the original article, you'll note that for at least a month the code has been receiving attention and refactoring from a team at Microsoft. Your criticism is that an academic team should have somehow convinced Microsoft to donate these resources before the epidemic became severe.
I'm going to go way out on the thinnest limb and propose that major software companies were not chomping at the bit to donate resources to epidemic modeling teams until very recently, and this isn't the fault of the epidemic modeling teams -- as much as you seem to think it is.
I'm also curious about why they brought in the people they did to do whatever cleaning up was done before publishing this code. There are several businesses in the UK that do this kind of work as a major part of their activity in one way or another. Current or former staff with that sort of background would have been the obvious people to tap for assistance. To the best of my knowledge, the people who have actually been identified as having helped have no such specialist background.
Granted, but when publishing, these need to be primary concerns. Source code, then, is a communications medium.
Many famous scientists like Einstein and Feynman would do their work in their heads, not by filling blackboards with equations. When they needed to publish, they'd write down equations because that is the medium of communications. You make it as simple as possible, so people can follow you. E=mc^2 is brilliant because it is so simple. Maxwell's equations got a lot more traction once they were simplified down to just 4.
Scientists using computers for research or analysis are welcome to use any messy code they want, but if they wish for others to follow their work, they need to clean it up to the standards of the industry.
It was beat into my head from a young age that research is only part of being a scientist. You also need to be able to communicate your ideas -- hence all of the writing class requirements. Scientists need to take this to heart, and if they wish to publish in terms of source code, to write source code that others can read.
When even professionals in the field have trouble verifying your work, you've got a problem. It's been a while since I worked in scientific computing, but I never saw a single published paper with a program where the program even matched the description in the paper it was attached to.
As a trivial historical example - an entire world superpower regularly and persistently engaged in and acted on large-scale unscientific motivated reasoning against its own self-interest and the problem with that was not that the best scientifically verifiable ideas available were inadequately communicated.
(The real problem with the worst case estimate is that it assumes people don't individually change their behavior in the face of a pandemic.)
If they are truly minor, they should not block the change.
I am guessing colleague tries to refute reasonability, which totally makes sense, because if in every review they have to fix minor unreasonable comments to progress, that stalls the thing completely.
https://cacm.acm.org/magazines/2010/9/98038-science-has-only...
Reminds me of "don't roll your own cryptography." Well, all the standard cryptographic algorithms were rolled by someone, and they had to have thrown out their first pancake too.
"They aren’t worried about it — not because this is a disaster they are trying to cover up, but because this is a routine bug that doesn’t really affect anything important."
If you have a bug you can never know the side effects that it can be causing.
So neither the article author nor the guys who wrote the covid sim understand complex systems.
It would do them much good to read a few works by Taleb. Their problem is that they don't know what they don't know, they think they know what they don't know, but they don't.
Also saying that it's "ok that scientific code is crappy formated/strutured" clearly demonstrates that these people, neither the author of the article, truly understand intuitively how quickly things can (and always do) go wrong with programs because they are complex beasts.
This reads as an acadmic without truly intuitive knowledge of how things can go badly defending other academics.
If a product is meant to put millions into quarantine it must be damn well tested and not a quick demo playground.
A good product can obviously evolve from a quick academic playground, but academics seem to finish a product fully so rarely that they don't understand that unless they test it severly thourougly that it will have bugs.
/s
/s
Academic "scientific software developers" (lol) never have to prove anything works in production long term in majority of cases. They make a one-off project and never touch it again. So this generlization stands.
And yes, I am saying that they are eggheads.
Because this article and the covid sim clearly shows that "scientific software developers" (lol) must not be trusted with real world-affecting systems without some other real world developer checking it.
The fact is that the covid sim is garbage and anyone who tries to defend it does not understand programming and complex systems well enough.
The closest thing we have to this in industry is bug reporting and software security evaluation, and this is sporadic at best. Most commercial software developers keep their code closed source and do everything they can to stifle public knowledge of bugs, even after they've been fixed.
This code has never been peer reviewed, ever. That is indisputable because Ferguson has claimed multiple times that the code is/was entirely undocumented, and nobody outside him and his few colleagues understands how it works. The moment it started being actually reviewed, people found a breathtaking number of bugs like out-of-bounds reads, typos in random number generator constants etc. Broken RNGs! Of all things I'd hope an expert in cryptography would be very worried by, a missing digit in an RNG constant should surely be one of them?
I've spent many years of my career working on highly technical software. I've worked in offices full of professional programmers where many also had postgraduate qualifications in STEM subjects. Those teams did indeed work on quite complex codebases for years.
Do you know what else they typically had? Testing infrastructure that was ahead of its time, careful validation of the mathematics being used before it was implemented, a reasonably clean and systematic software design, and good traceability from the mathematical code back to the underlying mathematics.
This is not rocket science. This is not even unusual if you work in this field. It's just what happens when you let professionals with an appropriate skill set do the job: programmers with technical backgrounds implement models constructed by subject matter experts.
Having one set of skills without the other is not a recipe for success.
Because if you cannot do those things, you're not really contributing anything of any quality.
You write as if you think what I'm describing is somehow unusual. It is not. I have probably worked with at least 30-40 well-qualified people just in my own career from whom any well-mixed team of say 3-5 developers could have done what we're talking about here within a relevant time frame. We are not talking about rocket science, just good software development practices being employed by experienced programmers with technical backgrounds who are used to implementing the models of subject matter experts. This isn't some magical unicorn skill set.