Covid-19 in Sweden

__

One reason why nobody doubted the model was that it was secret—always a smart move. When on March 16, 2020, Neil and his team published their paper, they did not publish their code. Yes, you read that correctly. Their paper made several policy recommendations based on predictions of a model they did not publish. So there was no way of knowing if their claims were true or false. They were simply baseless claims without any evidence.

To me, this destroys any trustworthiness Neil Ferguson might have had left. One hallmark of science is independent verifiability. Without others being able to verify the claims and run the model themselves, it is not science; it is pseudo-science with a strong appeal to authority.

On April 27, over one month after the paper in question was published, they finally released their code on GitHub. However, it turns out that they did not release the original code used to generate the predictions in their paper. The released version was edited by software engineers of GitHub to make it acceptable:

Before the GitHub team started working on the code it was a single 15k line C file that had been worked on for a decade, and some of the functions looked like they were machine translated from Fortran. There are some tropes about academic code that have grains of truth, but \

— John Carmack (@ID_AA_Carmack) April 27, 2020 As you can read in this Tweet, the original code was “a single 15k line C file that had been worked on for a decade.” Everyone with a little bit of knowledge about software engineering should be shocked. A single 15000 line C file is beyond bad practice.

But even GitHubs’s engineers could not salvage the code. The released version is still riddled with bugs and random equations that literally nobody can explain. Moreover, even if the released code were exactly the code used in the paper, no one could have replicated the results because the input parameters used in the paper were not published. As the GitHub page reads:

IMPORTANT: The parameter files are provided as a sample only and do not necessarily reflect runs used in published papers. At this point, we are long past science, with a small s, and far into the world of Science, with a capital S. The latter is the world where the term “Believe the Science” makes sense. While science (small s) is the process of doubting everything, disregarding authority, and searching for the truth, Science (capital S) is where the truth is determined by fiat, and your job is to believe, not question it. But I digress.

IV After several issues around the non-deterministic behavior of the model were brought up on GitHub, the team responded with an interesting answer:

We are aware of some small non-determinisms when using multiple threads to set up the network of people and places. (Look for the omp critical pragmas in the code). This has historically been considered acceptable because of the general stochastic nature of the model. Non-deterministic means that given the same input, you do not always get the same output (non-deterministic behavior is not necessarily bad; only in cases like this where it is not explainable). Their answer to this problem was that it does not matter because the model is “stochastic,” which is just a fancy for saying that they run the model multiple times and average over all the outcomes.

Every time a new issue around non-determinism came up or a different bug was discovered, the team’s answer was the same:

This isn’t a problem running the model in full as it is stochastic anyway. But is this really true? You do not have to worry about bugs because “it is stochastic anyway”? The answer is, obviously, no.

To understand this stochastic magic better, let’s take a look at an example: baking a cake. If we weigh flour, we might weigh it several times and then average over all these measurements—only if we are nerds and want to follow the recipe particularly closely, of course. But why does this give us a more accurate result? The answer is that if, for example, we make an error when reading from the scale (imagine an old analog scale), then an error in one direction (more) is as likely as an error in the other direction (less). In other words, if the flour weighs 100 grams, you are just as likely to mistake it for 102 g the first time, and 98 g the next. Only if we make mistakes sometimes in one direction and sometimes in another, averaging out works. Otherwise, it does not. If your scale is broken, and always shows five grams more, averaging does not help you.

The same is true for bugs. If, and only if, we could be sure that the bugs distort the output sometimes upwards and sometimes downwards (preferably with equal probability and magnitude), we could solve the problem by running the model “stochastically.” However, this is not the case because, by definition, we do not know how bugs affect the code; otherwise we would understand them and probably be able to fix them. The point is that nobody, including the “scientists” who produced this model, can know how those bugs affect the code.

Do not get me wrong, stochastic models are not necessarily like this. Most of the time, if the model is not too complicated (we will get to this point later), small changes in the input create small changes in the output, and the randomness in the output stems from intentionally included pseudo-randomness. However, this model is not non-deterministic in the predictable (explainable) mathematical sense. It is non-deterministic in the angry-toddler-in-a-toy-store sense: nobody knows what is going to happen, and there is no way to replicate it. Put differently:

It has nondeterministic outputs that do not follow from seeded pseudorandomness but are rather an inexplicable part of the process. I am not using “inexplicable” rhetorically here: nobody can explain this. This is one of the great issues in Complexity Science. Clearly there is a stark mathematical difference between deterministic and non-deterministic. But there is also a fuzzy, and arguably more important, difference between non-deterministic and really, really, really NON-DETERMINISTIC. Unfortunately, the Imperial Model falls into the latter category.