Why it’s so hard to make a good Covid-19 model | Hacker News Reader

Why it’s so hard to make a good Covid-19 model

fivethirtyeight.com

154 pointsgHeadphone85 comments

What this article misses is that simple models of complex systems in science are most useful for understanding the dynamics of phenomena, not for making accurate quantitative predictions. This is not a model of a mass accelerating in a vacuum where Newton's laws are sufficient to a high degree of accuracy, or even a numerical model of the aerodynamics of an airplane, where the physics are well understood but there are far too many particles to solve analytically. This is a parameterized model for the behavior of millions of people, each one's reaction to exposure to viral particles, legal and social norms, personal and economic situations, and so on. Of course we are not going to be able to predict the future of an unprecedented event like this quantitatively. It's not just that data are hard to obtain accurately; the models themselves are so simplified that they can't capture much of the important dynamics going on.

Agree. I think a better exposition on COVID-19 models is Zenep Tufecki's article in The Atlantic [1]. TLDR: They're more directional roadmaps, showing you which possible outcome paths you need to prune off the tree of outcomes, not exact numerical predictions.

[1] https://www.theatlantic.com/technology/archive/2020/04/coron...

In a sense, much of this work is similar to A* search and reinforcement learning. You have a limited view of the world, the responses to your actions are not directly causal, but if you can use the data you have to make reasoned decisions on what parts of the (extremely large) action space are unlikely to pay off, you can avoid wasting resources on them.

It's hard to use a roadmap with this much variance. Is there anything that can be said with certainty about covid

It's not the variance, it's the worst-case analysis that is helpful. You take decisions that lower the worst-case outcome.

Sure, there is lots that can be said about COVID-19. Epidemiology is a fairly well-studied field, and we have experience with aerosol-transmitted respiratory viruses. We have a pretty good feel for the transmissibility, getting better all the time. Pretty good feel for the disease course, and getting more specific about the different phases of treatment all the time. Pretty good feel for the R0 in many conditions, and the serial interval, and so on.

"Is there anything that can be said with certainty" is a dangerous position to be pushing on the public. It is the very thing that autocrats and dictators try to induce in populations through propaganda and disinformation, so that reason and self-help are surrendered. When there is no way to know truth, truth is what the Leader says it is. And we don't want to go there.

I don't think it makes sense to shoot for the least bad worst case. How about the best average case.

It's like trying to calculate the Butterfly Effect during winter at the Reserva de la Biosfera Mariposa Monarca (Monarch Butterfly Biosphere Reserve).

https://en.wikipedia.org/wiki/Butterfly_effect

https://en.wikipedia.org/wiki/Monarch_Butterfly_Biosphere_Re...

I don't think it is - I think you can write "x is like y" but you have to understand and explain why. Otherwise I can say "a hippo is like a bright blue sky" and then claim that I am right! I don't think that this is like modelling the impact of a butterfly on storms - I think this is more like modelling the diffusion of a bottle of dye in a swimming pool.

Grandparent is somewhat nonsensical, but I think what they were trying to say is that since human society is a highly dynamical system, it's not really possible to have a highly accurate model to represent how COVID-19 is going to spread. The best we can do is use naive models and plan around worst-case scenarios.

But we know really quite accurately how people en-mass move and associate prior to intervention. The proper models are using this information.

>the models themselves are so simplified that they can't capture much of the important dynamics going on

That isn't my (outsider) understanding, could you provide a reference? By reference I mean a recent survey that demonstrates the specific limitations?

My understanding of the Imperial College model is that it uses a 30m*30m grid of the world and the expected people in the world and various additions to simulate schools and workplaces. Maybe you know better?

30m is quite a lot bigger than the Planck length, and you can’t predict human behavior by playing The Sims.

you can predict human behavior with big data and linear models/embeddings with a few billion parameters. This works so unbelievably well, I expect that eventually health modelling folks will do this, provided they have enough high quality data.

Alice gives Bob a number, x, and tells Bob to calculate f(x), where f(x) is defined as the partial function with goedel index x applied to input x. She says that if or when Bob finishes, he should come bring her his answer and they'll get drinks together. Given parameter x, do Bob and Alice ever get drinks together?

Well, shit yus, because Bob has a dominant option to go and get Alice to have a drink if he has an answer or not. If Bob gets the answer Alice will be impressed. If he doesn't Alice like him anyway - so what's the odds? It's undecided what happens next.

Actually, it seems to me that more or less, that is exactly how some of the network based contact tracing models work:

"Therefore, characteristics of mixing networks—and how these deviate from the random-mixing norm—have become important applied concerns that may enhance the understanding and prediction of epidemic patterns and intervention measures." [1]

And as a follow-on, that is why there is so much discussion about using mobile apps for contact tracing - it builds the network for you, passively.

1 - https://royalsocietypublishing.org/doi/10.1098/rsif.2005.005...

But the purpose is not to predict individual behaviour. I find it really frustrating that people are completely happy to discount 100 years of epidemiology so casually.

Me too. It is also frustrating that they reject it on assumption on what it might take into account instead of trying to find out what it actually take into account.

Like, people in this forum assume models are done by data analysts with no special additional knowledge about domain.

You can hype up the craft all you want, but none of the models got anywhere close to what we're seeing in reality (look at the estimated number of hospital beds in the IHME model, for example). How can you explain that without admitting that the models were based on poor assumptions and/or data?

The models I have seen made model and then tested how it reacts as assumptions (or variables) changes.

Then there were models that specifically searched best case scenario or worst case scenario. The assumptions were clearly stated too, so you was able to determine whether you agree with then or not.

They made predictions about asymptomatic cases before those were actually measured. They made predictions before those hit the news.

They were also pretty open about unknowns and explicitely said what they are not trying to predict.

IMO the IHME/Murray model communication was really confusing, especially considering it immediately became the gold-standard model for the USA.

CT Bergstrom (UW bio prof in communication with IHME team) lays it out in his rapid peer review here: https://twitter.com/CT_Bergstrom/status/1243837050253496320

The uncertainty intervals are pretty confusing even to educated users trying to understand the results in good faith. The entire model used some version of the Wuhan intervention as a prior, and only the uncertainty in fitting curves to that prior is represented in the intervals.

The problem is that the overwhelming majority of the uncertainty in actual outcome doesn't come from the curve-fitting uncertainty, but rather what prior to fit a curve to.

The IHME model seems OK for what it does, but I'm baffled as to how it become the most-cited tool that we have. It's totally inflexible, and pretty confusing in terms of what it actually represents. Its overestimates are now being used as a bludgeon against people that take the virus seriously, which IMO is a major issue and I think wrong, but difficult to refute given how inflexible and confusing the model is.

I'm not happy to discount 100 years of epidemiology, and don't do so casually. But everything we've seen suggests that it can't produce accurate estimates of how bad a pandemic will be.

I don't mean to imply epidemiologists are bad at their jobs! Some areas of science are just like that. It's similarly hard to predict from first principles how effective a new drug will be or what properties a new material will have. But I've seen a lot of people say "we've gotta do suchandsuch because this model I saw projected it as the best option", and that's not a level of confidence these models can actually provide.

There is another point too, which also to some extend goes against your point—that we model epidemiology in standard ways, and that in order to use those standard ways we need the parameters in the set of differential equations. And these do make predictions based on the equilibria. I am not saying they solve everything, but they are routinely used to for example calculate how many people need to be vaccinated to stop outbreaks of measles.

The article does mention most of the parameters; but we don't know what values to assign to those parameters. The last scientific writing that I was reading speculated about the R_0, but I believe we are still not sure about that. In any case, the point of the parameters is to find R_0, so we cannot expect to have an accurate R_0 without them.

We also don't know mortality rates in a general population (look at Italy vs. China, when specifying to age groups). But anyway, all I wanted to say is that our models are simplistic, but surprisingly useful.

The point of the parent is you can't actually find R_0 because R_0 depends on the behaviors of millions of people which cumulatively are based on billions of factors. R_0 isn't a constant, it's an equation. Mathematically, unless you get really lucky or have a very idealized system, the standard models underfit the actual system.

The models can still be useful though, e.g. 'we must reduce r0 below r_critical to eliminate spontaneous spreading given herd immunity %'s' is worth knowing.

This is not as absolute as is being presented. We can speak about spreads and aggregate behavioral patterns just fine if we take times to infinity and zoom out far enough. The next few weeks or localized transmisson (which is "impossible" to model) are problematic, agreed.

Actuaries do a pretty good job predicting things at a population level, where many factors average.

R_0 changes as people change their behavior. It is not one constant common to all places and times with virus.

Edit: people down voting this, please look at definition and those papers. R_0 does changes.

Even without changing behavior, R0 can change as a virus enters different conditions. The flu in the winter (aka: Flu season) has higher R0 than the flu in the summer. Even if all humans acted the same in both cases.

R0 is assumed to be a constant so that the math / models work. But it really changes dramatically in practice. Maybe someone else will eventually make a model where the constant is more... constant. But until then, we will continue to use this model.

How certain of this are we (for the flu)?

When I've looked into it, I've found it difficult to assess the degree to which we are looking at "behavior changes between seasons" vs "the season matters".

Reasons the season might matter: Thicker air (higher humidity) may reduce particulate spread. Viral lifespan may be different under different temperatures.

Yet, in researching the issue, I tend to find things like "well, in animal models, this is what we've seen". Or "we think the heat reduces the effectiveness of a protective membrane that the flu uses".

Despite some effort, I've not found sources that both (1) seem reliable, and (2) provide a clear/concise understanding of what we know and why/how we think we know these things.

http://sitn.hms.harvard.edu/flash/2014/the-reason-for-the-se...

They stuck a bunch of guinea pigs into a box, one box at 41F, a 2nd at 68F, and a third at 86F.

The box with 41F had the highest transmission rate. The box at 68F and 86F had lower transmission rates, even 0% transmission rate in the presence of 80% humidity.

----------

No such experiment exists for COVID19 yet. But because COVID19 continues to exponentially grow in the southern hemisphere (Australia in particular), there doesn't seem to be a relationship between temperature, humidity, and COVID19 like there is with the flu.

What you said reminded me of Richard Feynman when he worked on the connection machine:

http://longnow.org/essays/richard-feynman-connection-machine...

The weather has far more moving parts than people (in aggregate). It’s the canonical example of chaos (theory). Until about a decade ago, assuming today’s weather is also tomorrow’s was still outperforming our prediction models.

But recently, we have become quite good at weather predictions 7 days out, and even for 14 days are now significantly better than chance.

Disease models can be useful even in the absence of predictive power. Getting any one of a few dozens assumption wrong can throw your prediction off by orders of magnitude. But it still allows you to, for example, explore how sensitive the epidemic is to different policy alternatives.

Besides: what are the alternatives? As long as you do anything, that action is based on some “model” of how the world works, how people behave, what value you assign to competing objectives. Writing that model-in-your-Head down or implementing it in software is strictly better than not doing so: it forces you to be explicit about the assumptions you make, it allows people to cooperate, it forces them to be specific in any criticism, it is a far better tool to communicate your reasoning to people affected by it, it deals in real numbers and will quickly expose any significant oversights you might otherwise miss, it’s accuracy can be measured and thereby improved...

Well said, but it should be noted that making quantitative predictions based on simple models (and scattershot data) of complex systems is FiveThirtyEight's bailiwick and Nate Silver's claim to fame. The subtext of the article is that the problem is so difficult even they are stumped.

They also would be limited from privileged data, like how many ICU beds are occupied by Covid-positive patients. The data requirements to pull off are high.

Its pretty simple: there isn't any decent data.

The data we have is strongly biased to older and sicker people.

There is no systematic surveillance of a geographic area, only panic testing of those who are showing symptoms.

Until there is sampling of a a borough, city or town, from start to finish, we will have wildly wrong models.

The only thing that we can plot reasonably accurately is the exponent of the fatalities, but even then its because its based on mostly hard data (unless its china...)

There have been a few of those samples.

https://www.cebm.net/covid-19/covid-19-what-proportion-are-a...

Note that this article is from March 31st, and while that wouldn't normally be very long ago things are moving extremely quickly.

In the 9 days since we still have models that are all over the place. Nate Silver covers the difficulty in predictive modeling of pandemics in his book "The Signal and the Noise" and in the years since it was written not much has changes to they're basically rehashing the points he made then.

Well, politically things are moving quickly too. Bottom line, all the “science” failed to engage The Cautionary Principle when they should have known that what they did know was likely outweighed by what they didn’t.

I can't tell what you think should have happened differently here

"Meanwhile, a report from Imperial College London that made headlines for its dire, modeling-based forecasts predicted about 2.2 million U.S. deaths from the coronavirus, if nobody changes their everyday behavior."

On the other hand, they're still quoting that study, which was very out of date well before the 31st.

Do you (or anyone else) have any resources for a critique/comment of that report? I'm interested since I tried reading it and I think I lacked a finer understanding.

I do not have any specific critiques about it, although I remember seeing the authors had some follow-up work.

But the important caveat about that model is that it assumes everyone does nothing. It has been a rather long time since everyone stopped doing nothing, so even if it is entirely correct, it has not been relevant since well before this article was written.

The main shift I've noticed (as a layman) is that people are thinking the illness is more contagious and less fatal than first thought. That explains the downward revisions in future deaths, but it also means that physical distancing is more effective and more important than first thought. Because a carrier that stays home is infecting ~5 less people rather than ~2 less people.

The thing I'm worried about today is population centers deciding to relax mitigation before they've vastly increased testing capacity.

Physical distancing is important if you are old (>65 years) or have a pre-existing condition.

The remainder should treat Corona as they do with the common flu, that is what the data is telling us.

References https://www.globalresearch.ca/swiss-doctor-covid-19/5707642

https://www.epicentro.iss.it/coronavirus/bollettino/Report-C... (Italian)

The article is a long listing of ways to say GIGO. It's hard to get good clean accurate data.

GIGO is a principle i learned at audio engineering school and it applies to all input/output relationships across multiple disciplines.

Have a 4k tv but only a 1080 cable box? Your tv will only show 1080.

Did you record too much compression on the master track? Now you have to deal with that in the mix.

GIGO is any time you get an input that will never give a desired or expected output. In circuits, its easy. It works or it doesn’t. In data science, GIGO can be hidden by hiding the assumptions made to reach the level of knowable information from the data provided.

Data scientists have a huge uphill battle right now. It is far more complex than looking at the numbers and trying to find patterns. People can find patterns in clouds.

Just this evening some Italian scientist announced on his twitter account that the number of new positive cases does not correspond to the number of new tests carried out and announced for that day, because those tests could have actually been made 2 or 3 days before or something like that.

Which, presumably, instantly invalidated all the charts and data-modelling based on the "number of new positive cases" / "total number of tests made" (with a lower value being seen as better).

But the Region of Sicily (or its official twitter account, anyway) replied that in their case the number of new positive cases and the total number of tests made are indeed correlated, which of course means that everything is a mess in terms of data coming in and its significance.

Later edit: For those who know Italian this is the tweet [1] I was writing about, and it looks like I was remembering wrong, the guy is not a scientist per se, more like a "data scientist", he seems to be working at a company very similar to fivethirtyeight (but presumably focused on the Italian market).

[1] https://twitter.com/lorepregliasco/status/124827958933764505...

That's right, difference between the date of specimen collection and date of announcement. General approach is to announce as soon as positive confirmation hit. There is a vintage to this data. Source: working in this dataspace as we speak.

Keep up the crowd-sourced wisdom HN, it helps prevent us data science folks in thick of it from keeping blinders on!

In jest, I am imagining a bunch of data scientists working late into the night, running the numbers and various scenarios, and then one person finally stands up and says, "FK it, let's just shut down the whole country and hope for the best."

That's totally unrealistic. A data scientist is never in the room where decisions are made

> Why it’s so hard to make a good Covid-19 model

Because there's no downside to the authors for overpredicting deaths and resource use, and _a lot_ of downside for underpredicting. So all the "bad" things get taken into account, and all the "good" things are ignored.

One thing I've reinforced in my view of the world is that common sense is very uncommon indeed. "2 million deaths" my ass. I hope people reconsider their trust in other models that are "hard to make".

2 Million deaths was without all the drastic measures we've taken. I'm having a hard time understanding what exactly you are suggesting we (society) should have done here. "Used our common sense" to do what?

Good luck trying to predict how much of a population is going to refuse to isolate and interact anyway.

Also relies on governments and politicians to not fudge testing and death counts, which is never going to be accurate.

btw the financial times has maybe the best graph on the stats however flawed:

http://com.ft.imagepublish.upp-prod-us.s3.amazonaws.com/2251...

The actual model used by UK imperial college estimated compliance with karantene of sick people 75% and compliance with general stay-at-home 50%.

What I find frustrating is that actual models used by epidemiologists back in february and March incorporated pretty much all "gotchas" non-epidemiologists discovered today. And it does not matter, because non-epidemiologists still assume they are first ones to ever discover them.

I've found this topic pretty interesting, and I've enjoyed trying my hand at it myself.

One of the things I've been playing with is Insight Maker https://insightmaker.com/ This site is a totally free platform where you can set up the kinds of simulations this article describes (stock and flow models). You can even specify your uncertainty in your baseline assumptions and run sensitivity analysis to see what the relative impact of each factor is on the model, and the range of potential outputs you could have. This system is very much like https://www.getguesstimate.com/, except much more flexible and way less intuitive.

Insight maker isn't a professional tool, it's really more of an advocacy and outreach platform, but despite that it's really quite powerful.

I think after you've read this article and internalized the difficulties in modeling pandemics (and have re-affirmed to yourself that you are not an epidemiologist, unless you are, more power to you if so), you might have some fun trying to build the model this article describes.

I'm still curious about the relationship between R0 and doubling period, because you can kind of see a relationship between the two by examining contagion period.

In general we've seen doubling periods that seem to suggest 5-6 days, but occasionally as fast as 3 days, unclear how distorted those numbers are by testing and mitigation and misattributed deaths.

If it has a natural R0 of 6, then it means that an infection will infect 6 others within that contagion period.

If we're thinking R0 is not 2-3 but is instead 5-6, then to make consistent with the doubling periods we are seeing, it'd mean that people are contagious for a longer period of time than we first thought.

I tried to do some exploration with the data to get some idea on the final number. One of the best plot I found that could tell the final number is plotting the percentage of cases in the next week with the cases till now. It is like negative half parabola and the time it meets the x axis will give the final number per country.

This is the final plot: https://i.imgur.com/5p4Xife.png. You could make a good guess from the data how many people it will affect in the lifetime per country by continuing the same pattern till it reaches x axis.

I really like the neher lab prediction tool.

https://neherlab.org/covid19

You can choose different scenarios and compare it with real data. So you can match the parameters of your model to the actual measured data (number of deaths) and it also has data about the population and the state of ICUs and hospitals that you can use in your prediction.

In the end it is just a tool which can give completely wrong predictions, but you get a felling for what could and could not happen.

The results of lockdown in Spain and Italy are not in this post and they provide crucial information. The lockdown in Spain has reduced in two weeks the daily infection rate from 42% to 4%, so there is hope for the future. But the problem is that the economy can't cope with the lockdown so we have a big problem. Also since the R0 can be reduced so much with political decisions, the emphasis should be on the political and economical ground.

Garbage in, garbage out! Can't build good models with data that's terrible. Inconsistent testing, different reporting methods, lying from governments.

Garbage in policy out

Completely tangent observation: What's the point of using sketch scribbles over the diagrams? They could just make it in powerpoint and simplify. It would be easier to read as well. Decoration for the sake of decoration? Why?

Charitably: To express the tentative nature of the information or model. "Here's our working theory."

(I think it makes it somewhat hard to read in this case)

I like the look of it.

Don’t all these models ignore the effects of warmer weather? A few studies have suggested there’s at least some effect.

We know it's hard Nate, it's hard for everyone else too.

That being said, I'd pay good money to hear his latest numbers...

It's novelty. Basically all models are based on assumptions in closed system. As one gets more information they add it to the closed system eventually making it easier to predict. As we know reality is extremely heterogeneous and an open system with too many interactions. This in principle makes all practically wrong. But it does not mean that they are not useful. (At least they are useful to make scary graphs that convince presidents).

I had a lame model that is currently predicting lower death rate but in a similar trend [1]. In this model my assumption is the lockdown to continue for 45 days. The result which I regret to have seen shows a scary number of 600k deaths after 39 days. So I'm hoping for something spectacular to happen such as vaccine, a drug, the sun etc that I would use to change my prediction.

[1] https://news.ycombinator.com/item?id=22814927

Because people have unrealistic expectations.

Best article I've read through this entire Pandemic.

What about our models for global warming?

Correct. Non-linear dynamic multivariate systems are very difficult to model and uncertainty is a function of time. That's why we can't predict the weather more than a few days out with any degree of accuracy.

Well those are obviously perfect. /s

Those two things are unrelated.

Unlike a virus, the laws of physics don’t appear to mutate.

Also, you can’t measure the quality of your model of virus progression with earth observation satellites or hundreds of thousands of years worth of ice core data.

Also, rather unfortunately in this case, the lack of meaningful action by world governments means that none of the climate models have to account for feedback caused by the world actually acting on the results of those the models.

It’s going to be fun to watch this flood of nonsense convince precisely nobody for the next few years.

Compared to a spherical cow model we could make as soon as we realized that CO2 the oceans absorbed didn't stay absorbed we've only done a little better. It's a complicated system, we don't know all the detail, and the calibration of the older models we've been able to robustly test at least just wasn't there.

Compared to constant model where the temperature stays the same both the modern sophisticated model and the spherical cow model do much, much better. For simple things like "how much infrared light and in what bands does CO2 absorb" we can measure that in a laboratory and have it nailed down.

The thing with climate is that you don't need to believe in the forecast. You could just look at the PAST results. Let's say just the last 10 years. And compare that to the last 200 years. May be that will teach you that something is going wrong and not going in the right direction.

The problem with that is how far back so you look for things to be “normal”? After all, most of North America was covered with ice at some point.

Most of North America was also covered by ocean at some points.

Neither of which is spectacularly good for the North American economy.

Very odd little diagrams to show model components in this article - why don't they use digraphs like everyone who knows how to do stats?

oh...

Who thought it would be easy. One person can decide to break quarantine and infect other people.

Reply on news.ycombinator.com