Imperial College London have released their Covid-19 epidemic simulation
github.com
github.com
I like the way InitModel() crashes (I think) if a global pointer called bmh (short for bitmap header) isn't first initialized by calling InitBMHead() from Bitmap.cpp. I guess it's obvious to academics with giant brains that InitModel() depends on a bitmap existing.
But it gets worse - the pointer is declared in CovidSim.cpp but Bitmap.cpp reaches across the module boundary to initialize it. Sweet.
Edit: I feel bad now. It's great that this code was made public. I should be encouraging that. I think academics writing code for things like this should probably take an undergrad course on software engineering. I think the one that taught me about Abstract Data Types would be most beneficial the author(s) of this code.
It's nice they aired their source code out for people to look at, least we can do is to critique without snark.
Then, you end up having to turn this 'style' of code into a product and to maintain it...
(In the end the crash was due to corrupted input data, so I fixed the data and stopped debugging)
the crash was due to corrupted input data
Sadly, academic constraints (time; results required the day before) prevented me from actually filing a bug report.
Of course, it sucks if it takes a few hours before you get your crash, that usually tricks you into thinking the software did something wrong and not you. Best is if it dies immediately upon reading the input, and if it does that with an error or a crash isn't crucial, though of course getting a clear error message is always better.
As a comparison, a collaboration internal tool I need to use will print a helpful usage message, and then segfault if you call it without any arguments...
Given your concerns, I would like to mention two things:
1) the code was originally a single source file, so I'm not surprised the module boundaries are imperfect - the code is many years old and the module boundaries have existed for only a few weeks.
2) this was not written by professional software engineers, but epidemiologists, and they were working with limited budget, time, and programming experience.
The original code as used to produce simulation results that were in turn used for reports/papers/policy should be made public.
Some code I've used from scientific papers in the past made a git tag at the time of publication, and said "Use version XXXX and the provided sample data to get the results in the paper", or something of the sort. This could allow review and improvements post-publication.
It's like saying that programmers are professional English writers, so comments and such should be written in perfect English.
IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately.
(yes I know I've been lucky)
> IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately.
I think the best way to tell if it is sound and implemented accurately is to first require that it is implemented as simply (to understand) as it can be. Even then, it is hard to reason about code. But without simplicity, the problem is ten times worse.
I'm tempted to go further and say that source code is formalized thought. I've written a number of simulators in the past, sometimes purely to help me understand a system more fully. There's something beautiful about describing the behaviour of a system in elegant code. When there's nothing left to remove I get much greater confidence that I've fully groked the system. I'd say this is the area where the "computers are like a bicycle for the mind" quote is the most true.
If the original code was a single file and you’ve ‘fixed bugs’ then what is on GitHub can in NO way be ‘largely what was written’
This code was used to lockdown an entire country - defending the code now based on who it was written by and under what constraints is disingenuous. It must have been known that there were these issues and they ought to have been fixed before using the output for something that had such huge societal impact.
Edit: Oh this is very old code, that could be an excuse.
There's also a (flagged) submission on HN discussing this[3] referencing [1].
[1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/
Carmack is OK with it, he's put his name to reviewing it. That's not to excuse the bad testing, but he hasn't thrown his hands up and run away. He worked with them constructively to make it possible for us to even see it! Be more like Carmack.
EDIT: Also be like this person: https://github.com/mrc-ide/covid-sim/issues/161 Helpful, constructive and the developers engaged with them.
Also, doing these generalizations groups together people with very questionable theories ("It's the 5G") with others that have more nuanced criticism.
Personally (and yes, I am a scientist) try to look up whatever is said in the media, either by journalists or experts, no matter if the results end up matching 100% what it is said (often it is less, and on some cases there is no match).
I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies.
Dangerous garbage.
Most HN users are just publicly preening. It's like a Mechanical Turk GPT2. I actually doubt they can write code.
Go click around GitHub. They screwed up a shuffle, there are uninitialised reads, RNG bugs, the works.
Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors...
Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions.
The worst problem this all out murder attempt can muster is that the code is hard to debug which frankly is should be the default assumption for all research code, not too persuasive. Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly.
A more interesting critique would be something along the lines of this: https://www.nicholaslewis.org/imperial-college-uk-covid-19-n... (however it's not by a subject matter expert so the problems they find might well be because the misunderstood some detail).
Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read both the code and the associated papers to judge this correctly.
Nothing presented clearly compromised the (supposed) reliability of the distributions produced, so the impact of these bugs beyond the inconvenience they add is unclear.
To be clear, it is certainly not true that removing these bugs will somehow prove that the model and its inputs themselves are correct.
I really wonder what it would take for some people to lose faith in epidemiology. Has this field ever predicted an epidemic correctly? Is there any level of bugginess that would yield the output of these teams unacceptable, to them?
Source? I haven't seen these specifically cited anywhere.
> floating point inaccuracies,
Combined with (safe) race conditions, this will cause non-determinism that would probably be considered OK.
In general: John Carmack looked at the code and thought it was OK for what it is (decade old simulation transpiled from Fortran at some point). Some ex-Google guy thinks its horrible.
I looked at the code myself briefly. I haven't formed a strong opinion about the code myself beyond "it's ugly and I don't want to work with it, glad it's not my problem." I am however objecting to some of the comments here that make it sound like it is obviously broken for reasons that they just don't understand.
I think a comment on the GitHub is relevant: https://github.com/mrc-ide/covid-sim/issues/175#issuecomment...
> To add to this, please read report 9 properly. The 500k UK prediction was if governments did nothing whatsoever - we never believed governments would do nothing but we modelled it as a base case, because that's part of what you do when you model.
> With the full social distancing the report suggestd it might be possible to reduce deaths perhaps to 20k - a death count we have already exceeded. Nobody here is laughing about that. The report was also very frank about the uncertainty involved in trying to predict what might happen at that stage
Reliable and reproducible. I think this discussion, even if the code ends up being correct and the model OK, is worth having. It might not change anything today, but could set new (hopefully better!) standards tomorrow.
That said, that "personal level" statement is absolutely out of the line.
https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi...
That stuff is priceless:
else if (argv[i][1] == 'C' && argv[i][2] == 'L' && argv[i][3] == 'P' && argv[i][4] == '1' && argv[i][5] == ':')
I guess string comparisons are complicated. I also fail to see why they used ':' as the separator and why they didn't use a proper library to parse argv...
Still hurts to read though :P
Perhaps the real reason is the code typically smells of Swiss cheese.
From now on I vow to trust NO academic results of computer models unless source code is published along with instructions on how to reproduce outputs (or at least similar output!)
I bet the quality of local administration would go through the roof if all mayoral candidates were forced to be proficient at SimCity.
Given the complexity, poor language/framework choice (should have used either Rust or Tensorflow), bad code design, and unclear determination of the parameters (especially the fact they don't seem to be estimated from real data with Bayesian inference, or if they are they didn't release that), as well as the ludicrous CPU and RAM usage making it impossible for most people to run it and thus check it, it doesn't seem like it has any chance of actually being a good model.
If this is the state of the art for the most important statistical modelling project in the world, we are in the dark ages.
[1].https://www.express.co.uk/comment/columnists/frederick-forsy...
https://www.theguardian.com/world/2005/sep/30/birdflu.jamess...
He said that if it was the same as the 1918 flu, you could probably scale it up to 200M people. He didn't predict 200M, he speculated that it could be that bad.
And he wasn't alone - "A global influenza pandemic is imminent and will kill up to 150 million people" said "David Nabarro, one of the most senior public health experts at the World Health Organisation"
"A Department of Health contingency plan states anywhere that there could be between 21,500 and 709,000 deaths in Britain."
An unnamed WHO spokeswoman said "best case scenario" would be 7.4 million deaths globally.
https://www.newscientist.com/article/dn7787-flu-pandemic-let...
"And yet, the models show, if targeted action is taken within a critical three-week window, an outbreak could be limited to fewer than 100 individuals within two months."
Oh, he also showed how to react and keep the number of casualties to a minimum.
Don't believe what you read in dirt rags.