Analysis of the Imperial College Epidemiological Model

This is a completely unfair criticism. When the relevant decisions were being made two months ago, a huge number of influential people in the UK were seriously proposing to do nothing to get herd immunity. This model, which intentionally depicts what would happen if people indeed did nothing, was a big part of why people changed their minds.

You can't say the model is wrong because people would take COVID seriously anyway, when the model is half the reason people take it seriously in the first place. We're somehow forgetting history that's only months old now.

Where is this from? Who originally wrote it? Is this text’s origin really a random Facebook post, from a pseudonymous author with a cartoon profile picture and no claim of any serious credentials in the subject at hand? (Whether epidemiology or anything else)

Regardless of the merits of the text’s post (which I do not claim to be able to judge) all evidence has to be analysed for context as well as content. Simple “common sense” claims (with a couple of big words to impress non-epidemiologists like me) are made to debunk the models: where is the evidence rather than rhetoric, even some basic citations, and/or examples of or links to counter-modelling? The post doesn’t even link to the original model files from Imperial that they’re claiming to critique.

It’s perfectly _possible_ that the claims made in this Facebook post are correct, but it doesn’t mean anyone should take this post (and its conclusions) remotely seriously without asking some very robust questions of it.

The model was released on the 16th of March. Large parts of the world had already "locked down" prior to this date.

edit: of course there is no doubt this model was influential in other countries' decision to lock down.

https://twitter.com/ID_AA_Carmack/status/1254872368763277313

> The social distancing policies are being compared to a zombie alternative that cannot happen and will not happen.

This is absoutely correct, my company made us work from home signficantly before the lock down because they were quite rightly worried about business continuity. In the Imperial model, that sort of thing just isn't accounted for. We literally cannot say whether the natural societal reaction would've reduced that base case by 1% or 99%.

The second point I would make is that epidemics aren't different every time. Or rather, the underlying fundamentals of how diseases spread aren't that different every time. So how come, all of these institutions that are meant to be studying the spread of diseases are putting out papers based on hacked together research code from the 1980s? Like, what were you doing in the 17 years since SARS?

Where are the dozens of open source, high quality, back tested, parameterizable models sitting on the shelf ready to go?

In many ways putting out the results of some trashy model that you haven't even properly tested and trying to shape public policy with it is actually quite a damaging thing to do. Not only is it likely to lead to bad decision making, it's also going to destroy trust in institutions that desparately need to be trusted in these situations.

Scientific publishing is broken. Is not only about the methodologies and how some reviews are conducted, but you can't get the files needed to reproduce them, not to mention datasets that are almost impossible to obtain.

I've worked briefly with simulations, I think most of the items here are valid. The RNG especially is a very easy place to shoot yourself in the foot, especially if your main RNG is used to derive random numbers with different distributions, and your numbers might look off and you don't know why.

Example, your RNG is biased or does not have all the range/precision you think it has (similar to throwing a die when you are expecting a number between 1 and 10 for example - this sounds stupid, but think for example, your results come as 0-100 but biased towards smaller numbers, it will be a pain to find)

Now, some other criticisms were pointing to a potential multithreading problem in the model, which sounds like a worrying problem.

-Is actually familiar with academic code so that the usual pearl clutching can be tempered with realism. That doesn't mean throwing up one's hands and just accepting that all academic code is like this, but it does mean coming from a place of knowing what normal academic code looks like (much worse than this).

-Is actually familiar with production forecasting in other contexts (outside of the small bubble of SV) the writer says they have "done some model review of financial models". I have done a lot of such reviews and many models are implemented in idiosyncratic Excel. Yes, models that drive policy decisions in government and large transactions in finance, models that are used in internal forecasting in many places. All Excel.

A recognition of the difference between should and is. There are claims here about what should be done in modelling which are correct but with little recognition that they almost never are.

Really I would like to see a review by someone who is familiar enough with epi modelling and with modern software engineering practices to give a balanced assessment.

The only material criticism of the model that isn't just "this should have been implemented better" is that it assumes that people will not voluntarily implement forms of social distancing when many people are dying and that this distorts upwards the death count. An agent based model would model people's time varying contact matrices based on what they can observe (so as death rates go up, voluntary social distancing goes up as well).

Presumably this leads to a higher peak than mandatory measures because deaths lag measures which are visible to policy makers but not to the public, therefore people will distance later.

This is indeed not addressed in the model. It could be addressed exogenously by making interactions dependent on the rolling average death count of the last several days as that is probably the signal that will the cause the most widespread panic.

However. There is some important context here which is that:

-This was not even the only model used by SAGE. The LSHTM model was run alongside it and gave similar results.

-The "no intervention" case was sense checked using the basic equations of epidemiology.

Models that drive decisions can be criticised on technical-procedural grounds (ideally this model should use fewer globals, and use psuedorandomness better so that it gives the exact same result on every run rather than more-or-less the same result on every run) and on model concept basis (the model assumes zombies) but the most important assessment of real models is how they drive decisions.

In that regard the model was successful and drove the same decision as the very different LSHTM model, the models used by almost every other country, and what could be estimated using Epi 101.

That is the necessity of mass social distancing. Even the Swedes, incidentally, accept this. They have chosen to make everything except for the closure of the highest risk establishments voluntary. I wonder though if isn't the case that these restrictions are in practice voluntary in many cases even in the UK. I could easily visit my family and friends - the police would never know. My employer could continue having us come into work - there is no absolute ban on office workers being required to come in, just guidance that people who can work from home should. The actual legislation can be considered not only as enforceable rules but also as high-cost signalling. There will be people who simply will not take it seriously otherwise on the grounds that "if it were really that bad, they would ban it"

So to come back to the one substantive criticism - what different advice would a model that implemented behaviour changes have led to? I don't think it would have changed anything, if you tell policymakers: "listen, we don't necessarily need to make these things mandatory, once the coffins pile up people will stay home by themselves" what do you think that is going to drive? They're hardly likely to decide that we can leave off the restrictions. We came up to 80% or so of ICU capacity as it was. That tells me that anything that led to less or later social distancing would have put us in the position that Bergamo or Madrid were - at least localised and temporary overwhelming of hospitals. Potentially much worse.

Must be awful for the poor folks whose work is being savaged by everyone.