The Economist's excess deaths model | Hacker News Reader

The Economist's excess deaths model

github.com

243 pointsraldu141 comments

No matter the quality of the code presented here, thanks to The Economist for posting this! We all really appreciate it, even if we don't always seem like it :)

Even if it seems justified to shit on the code as is (bad style, lack of comments, no tests, whatever), all it does is discourage similar companies / researchers / folks in academia from posting their code as well. So then we end up with the same code, but now no one can see it - which is the worst of all worlds.

Let's support / work towards a culture of sharing our code before anything else. And if there's something you think can be improved, maybe consider opening a PR with improvements to really _show the value_ to all these parties of posting their code!

Amen!

When we have the code, we can all work together to improve the model, add the tests, find the bugs.

And honestly, research shouldn't be considered valid if all of the code and data isn't available. Reproducibility is one of the pillars of science, and that's not possible without code and data.

The code is not even that bad. It's a simple script generating a plot. I think it's too much to expect tests and documentation for it.

I took a quick look, and I found it readable and easy to follow, contrary to 98% of the codebase I see on github.

> Even if it seems justified to shit on the code as is (bad style, lack of comments, no tests, whatever), all it does is discourage similar companies / researchers / folks in academia from posting their code as well.

I agree that code should be published alongside papers, and great that the Economist, being a newspaper, publishes their code too.

But when it is the most reputable sources (researchers, universities etc) publishing code of quality so low cannot possibly be correct (looking at you Imperial), it totally should be scrutinised and critiqued.

I think the root cause is, where research is code-based (I guess 90% of science), it should pass the most basic correctness tests before a paper is accepted, or indeed turned into national policy. There are many angles to bite this apple from, but what remains unacceptable is shit code being taken at face value.

> publishing code of quality so low cannot possibly be correct (looking at you Imperial)

If this is referring to the quality of the model developed by Neil Ferguson @ Imperial College, John Carmack audited the code and didn't have a big problem with it: https://twitter.com/ID_AA_Carmack/status/1254872368763277313

I would love to see the tests that people come up for this...

Tests can be helpful and informative, or they can be make-work boilerplate that will never reveal any flaws. And in this type of code base, it's hard to imagine the test that's written in code that will be informative.

When dealing with data, "testing" is most often done visually, with plotting.

The process of exploratory data analysis is not anything at all like writing code to an architected specification, and the same practices are not as effective for different types of work. Insisting that there should be automated tests for something like this reminds me of the cultural phenomenon of extreme table-phobia in the 2000s that happened as a reaction to using tables for layout. I had trouble getting people to believe that is was OK to use tables for tabular data, it was extremely frustrating. Or in the last 5-10 years, with design trends preferring spare layouts with lots of empty space, it was super hard to get information dense designs into production, because they were so averse to the idea that people may need information density for some things.

I'm also reminded of this video about the deficiencies of notebooks:

https://youtu.be/7jiPeIFXb6U

Yes, there are huge deficiencies to notebooks. But using them is an entirely different task than the type of code that one would write in an IDE, for code that gets put into a library and will be run by others. I have two work modes: 1) IDE (actually Emacs) where I write tests and the code is meant to be reusable, even if it's a one-off script, then 2) exploratory data analysis and processing, in notebooks. Though both of these use code, they do not and should not share much else in the way of practices.

We develop best practices in fields to work against our worst instincts. And once we establish best practices, we become strict about them to make sure we don't fall back into bad practices. But when approaching a different field it's good to reevaluate these best practices to see if they still fit.

I think it always makes sense to have an oracle, or a few. Nothing super fancy, just a few values you _know_ have a certain outcome.

   f(4,5) == 6

is good enough. just, whatever you type in to test as you develop the code, write it down and save it for later. I mean, yeah, you can go nuts with boilerplate. But, I always seem to regret not having written down a handful of cases. If nothing else, it helps me remember what the heck this function is for, and a few samples of expected input and output, along with source are more immediately useful (to me) than comments.

Opinions differ. This isn't a hill I'll die on - as evidenced by my lack of tests from time to time. But I've never regretted the few minutes to provide them, and I have regretted not providing them

That sort of test doesn't really apply here in any meaningful way.

Suppose you want to validate that a certain column is in the file. You could place your assumptions around that explicitly in the read_tsv function call and have it generate the error for you on reading the file, or it can generate an error when you try to use that column. Either way, there's no "test" yet the intention is clearly stated in the code, and the code will simply fail out rather than proceed if it encounters input that can't be meaningfully used.

This code is in many ways a declarative use of other code, describing the data, doing some joins and filtering, and plugging it into well tested libraries of code. The data structures that store the data and the library functions that deal with the data are designed in a way that bugs often result in errors immediately.

In many ways it's like writing a SQL statement. One could make test data tables and make sure that the SQL statement does what one expects, but the SQL statement already embodies intent in the same way that a test of code would embody intent.

So, I'm not an R guy, but I pawed through some of the code. I get what you're saying, there's an environment, it needs to be just so, or it won't work. that's fine. And yeah, it does feel a lot like SQL.

I would point out this guy - https://github.com/TheEconomist/covid-19-the-economist-globa...

that sort of has the look of something that took a pass or two to get right. And I think, would be nice to have a test case (sample call).

Again, not a hill I'm going to die on.

Overall, I don't think we have a big difference of opinion, yeah, it looks like gluing together libraries - as a non, native R speaker, I think I can make sense of the project, it's cool they put it out there, and I don't think there's anything _wrong_ with what they've got.

That function is pretty standard.

If you want to run a loess, it runs the loess function if not it runs a windowed average.

What exactly would you test it for?

This is actually one case where I could see jfoutz's point!

If I were developing this, I'd probably be running it in a notebook, and inspecting output on a few rows from the data frame. It would be easy enough to capture both the input and output and throw it into a stopifnot to record the testing that was performed interactively.

I haven’t looked at the code but this stands out to me:

> When dealing with data, "testing" is most often done visually, with plotting.

Even if your validation process is entirely manual, you still can benefit from automated testing. Snapshot tests are completely agnostic from whatever serialization format. If you eyeball check something and say “yup this is good”, that’s something you could snapshot and never test again.

Data should only be taken seriously when the full context is shared for reference. Somehow internet created a generation of TLDRs or ELI5 which just is very sensible to lies and manipulation. Even though I can build trust to my sources, I still think it is mandatory to have access to the raw data.

The Economist has been publishing since 1843. I'd imagine that for the vast majority of that time, readers have trusted them with either no details at all about methodology or little more than a high level overview of what was done— certainly there was no expectation of being able to reproduce and iterate on the full analysis locally, even thirty years ago, much less in 19th century.

Don't get me wrong— this is a good direction to be pushing in. But it's something that's been very much enabled by the internet, which is why I'm finding it weird to be blaming the prior status quo on internet culture attention span memes.

> Somehow internet created a generation of TLDRs or ELI5

No, it has created a generation. of conspiracy-mongering freaks.

Or, more precisely, it has allowed that particularly seedy underbelly of society to become more noticeable.

The code doesn’t even matter in this instance.

The discussion should be on the model design and whether it’s sound.

Leave it to HN to start criticizing code whenever anything is posted.

The code absolutely does matter, as we need to be confident that it is a correct implementation of the model. To turn your first sentence around, if the code is incorrect than the model doesn't even matter.

I've done my share of implementing statistical models in code and have seen plenty of examples of incorrect code failing to implement the model correctly.

Your second paragraph doesn’t match the first.

Simply clone the repo, download the data, and verify the results.

The discussion needs to not be on the code but the quality of the assumptions in their model.

Which you can read here.

https://www.economist.com/graphic-detail/2021/05/13/how-we-e...

Why should I care about the assumptions of a model if that's not the model that is implemented, because the code contains errors?

Obviously the assumptions of the model are important. But I was responding to a claim that the quality of the code is not important, which is also obviously false. Neither discussion alone is sufficient.

Code like this is like (not) caring about memory leaks in a Missile Control module where the explosive is the garbage collector:

When code is one shot, accomplishes its goal, and does not need to be extended or relied upon in the future, why do all these things?

Is there any goal above that is not also serviced, at least in significant part, by open-sourcing it?

Bravo to The Economist for releasing it. This is unabashedly a Good Thing.

Some muddled thinking up in this thread. Testing is good for code implementation but the risk for code such as this mostly lies in untestable aspects like the assumptions built into certain types of statistical routines and measurement/definition problems. The only answer for these issues is independent replication. Software thinking emphasizes reusing code, but science should want the opposite. Replication really matters, even if we don’t want to accept that inconvenient fact. Great work with perfect code will fail to replicate and not because there were problems with the initial work, it’s statistically certain.

Recent and related:

There have been 7M-13M excess deaths worldwide during the pandemic - https://news.ycombinator.com/item?id=27177503 - May 2021 (457 comments)

It would be interesting if there are fewer deaths than expected in the next few years.

If it "averages out", that still means millions of life-years lost.

the point with death is that once you're dead it's over. and it won't "average out" in the sense of life expectancy. it gives the life expectancy growth a dent. it will only average out in the sense that everybody alife will die some day.

I wonder if they mean like the thing like what people say about DST causing deaths. But if you look on the large scale that blip does not exist. Standupmaths did a segment on it a few years ago. Basically the idea was 'yes you were going to die, and DST seemed to make it happen' but it seemed to only make it happen a few days sooner than it normally would have. Maybe that is what they are talking about? I personally know a couple of people who were marked as dying from covid. But the reality was they were going to die very soon. They both had very advanced stages of Alzheimer's.

as long as it's very.clear that the age distribution of surplus mortality in times of covid is heavily drawing from other than Alzheimer's dead-soon-anyhow; as long as it's very clear that covid-19 kills otherwise healthy diabetics, overweights, heart patients, cant-afford-healthcare people, was-misguided-about-indoor-events people, multiple sclerosis patients, the list goes on, and on and on, ....

as long as that is crystal clear, that "two old dead soon anyhow Alzheimer's patients" completely misrepresent the 500.000+ dead US citizens...

... such an anecdote is perfectly fine ...

There have been a number of studies (e.g. [1], [2], [3]) published estimating how many years of life have been lost to COVID-19. The estimates vary between ~10-15 average years lost per COVID death.

[1] https://www.nature.com/articles/s41598-021-83040-3

[2] https://www.news-medical.net/news/20201021/More-than-25-mill...

[3] https://www.sciencedaily.com/releases/2020/09/200923124557.h...

thank you for these! two us centric and one global, same results!

I can't resist: males of america! you are the the victims, the "loss leaders" of the plague:

> We find that over 20.5 million years of life have been lost to COVID-19 globally. As of January 6, 2021, YLL in heavily affected countries are 2–9 times the average seasonal influenza; three quarters of the YLL result from deaths in ages below 75 and almost a third from deaths below 55; and men have lost 45% more life years than women.

let that sink in:

> men have lost 45% more life years than women.

mask up! vaccine up! and don't let them mn-killers tell you otherwise.

/scnr

[Sorry for posting here. The other thread was flagged and subsequently closed]

Can you suggest me an advanced Bayesian Statistics book that focuses on application without sacrificing too much mathematical rigor?

I am graduating in MS in Stat. I've took a Bayesian Stat course that followed Statistical Rethinking by Richard McElreath. I liked this book because the author appeals to intuition instead of mathematical rigor. I took 2 semester long statistical inference course, so I am ready for some advance material.

Bayesian Data Analysis by Gelman et al. There's a free PDF available on Gelman's website [1]. Definitely application-focused, and the authors are a who's-who of contemporary Bayesian statisticians.

[1] http://www.stat.columbia.edu/~gelman/book/

Thanks! I'll check it out.

If this model works as expected, then it should take into account those who "would have die very soon" otherwise. It models the deaths that would have happened if it were not for COVID and measures the difference against that number. So the person you mentioned should be counted in the projection.

It shouldn't, because expected deaths in a time period should be proportional to the living population. If half your population dies in a war, your model should predict half as many deaths the following year, unless you don't update the model.

That's only true if a random sample of the population dies. If we assume (and I'm making this up) that in the steady state, the elderly comprise 90% of deaths, then if a war kills only young people, you'll expect a substantial increase in proportional death rate.

I was simplifying, the point is that the model should always reflect the current state of the population. You shouldn't expect the kind of "catch up" effects that OP is referring to, unless you have static predictions that don't take into account actual deaths and births.

But... you would. If all elderly people dropped dead right now, deaths per year would plummet for decades.

And your model would reflect that after the day all the old people died. So you wouldn't see any negative excess deaths (= actual deaths - predicted deaths). Otherwise you would see massive excess deaths from the baby boomers getting old for example, or negative excess deaths from the WW2 generation in places where many young people died.

In the long run we’ll all be dead.

So if you killed everyone on earth the mortality rate for the next eternity would be 0.

Please don’t tell the singularity

Mortality rate would be undefined (0/0)

;)

There will still be years of life lost per dead person.

We'll all be dead eventually, but most of us aren't OK with losing X years of it.

Possibly. This is known as mortality displacement.

https://en.wikipedia.org/wiki/Mortality_displacement

Looking at the official numbers in the UK, deaths seem to fall below the expected levels for a week or so, following a big spike in covid deaths.

It might… but it would still indicate that many living-years had been lost.

I don't know what fancy stats the economist does (and I don't trust fancy stats), but I looked at the french stats [1] recently and my back of the envelope approach [2] suggest 50k excess deaths. There was 55.7k more deaths in 2020 vs 2019 (which didn't seem to be an extraordinary year). Over a 25y period, the number of deaths increases by 3.4k every year on average (population growth).

The number of cumulative covid deaths in france is 109k, of which 65k to the 1st of Jan 2021 [3]. So the excess deaths seem to be in line with covid deaths.

Now you could get a very different result for another country depending on how well covid deaths are reported. Also difficult to predict how this will look like in 2021. Covid deaths seem to be predominantly people near their end of life, but on the other hand the delay of lots of medical procedures as the result of the lockdown should have its own excess death, plus impact of change in crime, accidents, increase in poverty, etc. God knows what will be the net effect of all this.

[1] https://www.insee.fr/fr/statistiques/2383440

[2] https://zbpublic.blob.core.windows.net/public/excessdeathfr....

[3] https://www.statista.com/statistics/1103422/coronavirus-fran...

In the USA, I heard it reported a couple of days ago that the nosedive in cases and hospitalizations, etc. is being attributed to a high jab rate of those 65 & up...because they were the primary group being hospitalized and dying.

The point being, excess deaths might be better stated as premature deaths. Mind you, death is death. But someone further along in life already nearing death's door isn't the same as a 25 y/o.

Seems that it'll be a year or more before we have more data and can quantify how premature the premature deaths were.

There's a horrifying subtext here, which is that if people who would have died next year died this year instead, that's... "OK", and it means that their early death doesn't count?

> It would be interesting if there are fewer deaths than expected in the next few years.

There should be, due to the “harvesting effect”.

https://en.wikipedia.org/wiki/Excess_death

Lots of comments around code quality, primarily as it relates to testing.

The real code quality sin here is that R, a vectorized-by-default language, is being used to employ for- and while- loops everywhere. It’s strange to see given the absolute plethora of R resources available aimed at avoiding just that.

But props to the economist for releasing the code, anyway. I’m a happy subscriber and will continue to be one.

Well, they only started using R a year or so ago, so they can be forgiven.

> Until fairly recently, we were less comfortable with statistical software (like R) that allows more sophisticated visualisations.

https://medium.economist.com/mistakes-weve-drawn-a-few-8cdd8...

That makes it even more strange, as all modern (ie within last 5 years) resources push the tidyverse.

2017 had more excess deaths than 2020[1]. The DALY loss was much greater too. What happened in 2017 that dwarfed the pandemic?

[1] https://www.pnas.org/content/118/16/e2024850118

Flu? That's my completely uneducated guess

What happened in 2017 that was completely unresponded to that dwarfed the pandemic when the world shut down.

If I'm reading this correctly, this paper is defining excess deaths as the number of people who died in the US who wouldn't have were those people living in Europe instead of the US at that point in time? Which points to structural issues in the US system, but not to some event in 2017 that resulted in extra deaths vs. any surrounding years (just that the death rate in the US has dropped much slower than in Europe over the study timeline 2000-2017).

The US death rate didn't appreciably increase (~3 per 100k) in 2017 vs. 2016[1] and has generally been improving. That's about 10k extra deaths, not 400k, like the paper says (because the paper's usage of that term is very different than The Economist's!)

https://www.cdc.gov/nchs/data-visualization/mortality-trends...

> but not to some event in 2017 that resulted in extra deaths vs. any surrounding years

But that's exactly what the paper shows. It's reasonable to ask the question why was the difference in deaths far greater in 2017 than in the surrounding years and why was the mean age of the persons excessively dying so relatively young? There most likely are structural differences causing excess US deaths, but the claim that those structural differences caused a huge relative spike in one year for no particular reason isn't very satisfying.

Did you read the paper? It effectively has three US data points it's using: deaths in 2000, 2010, and 2017. And it's comparing to European countries in the same time to calculate excess deaths. But they're defining excess deaths as those due to living in the US vs. Europe. Not excess deaths in 2017 vs. 2000 or 2010. This is pointing to structural issues in the US (like healthcare, violence, opioid issues, etc.) causing a long-term mortality rate difference between the US and Europe (it's significantly worse in 2010 as well as 2017). Not that there was a peak in 2017.

I don’t see how I can state what I’m saying more clearly and you’re obviously not following. Maybe try reading the comment you’re replying to more carefully?

I got something different from that paper. Quote from the abstract:

"Applying European age-specific death rates in 2017 to the US population, we then show that adverse mortality conditions in the United States resulted in 400,700 excess deaths that year."

To paraphrase: on average in 2017, US people died more than the equivalent people in some bit of Europe.

No great event happened.

It’s still significant because it’s a huge anomaly to the annual mean difference between the two populations.

Absolutely. But the paper is talking about a long-term mortality divergence between populations, not a peak in 2017 like you were claiming (and there's nothing about 2020 data in this paper).

There was a recent year that flu vaccines were remarkably ineffective. At least here in Australia.

Perhaps that was worldwide?

I haven't gone through the data myself, and maybe someone else already mentioned this, but I am not sure whether "excess deaths" takes into account lifespan. It has been roughly 70 to 80 years since "baby boomers" were born; based on common lifespans, wouldn't it be expected to have a "death boom" around this time?

You can estimate gradual changes like that by looking at a long-term graph.

Here's a graph from before the pandemic, including a (now out of date) projection:

https://www.macrotrends.net/countries/USA/united-states/deat...

(Note that the slope looks exaggerated because the baseline isn't zero.)

The general way to do that correction is to look at actuarial tables for deaths by cohort. These models are obviously not 100% accurate, but they do take into account many factors, and are incentivised to be as accurate as possible as they determine the profitavility of life insurance.

In this context, excess deaths are variations from these tables - more people from a cohort dying in a certain time period than expected.

Is this estimating the right quantity? Wouldn't it be more relevant to estimate the total number of years of human life lost due to covid, rather than the number of deaths attributable to covid?

I’d imagine that’s a lot more difficult.

It would be more difficult, but I don't think it has to be "a lot more difficult". The code could make a lookup in the actuarial life tables for each country to get the expected remaining years of life for a person of that age, and then just aggregate these, no?

Ballpark yes, but that's likely a huge overestimate, as even normalized to age, its people with co-morbidities that have the lion's share of deaths.

Yeah, you can of course keep approximating it. The next adjustment would then probably be to use something like quality-adjusted-life-years (QALY) based on the persons co-morbidity, and then (if you want to) your could also take it the other way and reduce QALY for survivors with long-covid.

Definitely an interesting problem though.

You'd probably want to gather life expectancy rates for individuals once they reached five years of age (since infant mortality necessarily lowers life expectancy but after a cliff people who make it tend to live longer than the average with a bunch of 0s, 1s, and 2s in it) for each country in question.

Then you'd need stats for the age of the dead in each country which I guess you don't really have in most cases since the deaths aren't aggregated anywhere since they haven't been attributed to COVID.

Thanks to the Economist for publishing their model. Would be neat to see more stuff like this. I haven't delved too deeply into the code (several 1000 line scripts which would take some sussing out) but it's nice people will have real world projects they can delve into.

You could make an argument that society as a whole should start looking at things that way, but personally I'd have no basis for understanding how that translated to damage. 3 million people dead is a number that I have a lot of mental models around, it's easy for me to get a sense of magnitude that way. 50 million human years lost is a lot less relatable number and I'm not sure how to contextualize it.

It might also be interesting to measure the cost of the pandemic in terms of economic harm, how much the GDP dropped, how much production was lost, how many stores closed, etc.

It's easy to contextualize. 50 million human years lost means that people would have to put in 50 million human years worth of work to recover the economic loss of COVID-19.

Consider the average human worker. In their life, they will output around 10.5 human years worth of work. 50 million years represents the total output of about 4.76 million human lives.

So if you delete the production of 3 million people due to COVID deaths plus all the time spent in lockdowns, 50 million human years lost sounds about right, but probably could be higher. I'd say it's probably about 80 million human years lost.

Perhaps the value of a human life is that they are alive, not the hours of work they produce during that lifetime.

Valuing people as primarily their economic output is anti-social to the point of being dystopian, and we have much better economic indicators than arbitrary heuristics when we do want to understand the economic damage of the pandemic.

Right. Think of this as "50 million years of opportunities for grandchildren to get to know their grandparents or 50 million years of time spent as a widow after your elderly husband dies" as well as just the economic cost.

This assumes that the lost human years are productive years, which I think is dubious at best given the demographics of Covid death.

We could count "years of productive work lost", but that feels like a very cynical way to look at it.

> 50 million human years lost is a lot less relatable number and I'm not sure how to contextualize it.

On way of contextualizing it that doesn't equate lives with economic output is to divide by life expectancy: if life expectancy is 70 years then 50 million years lost ~= 0.7 million human lives that never happened, or 0.7 million healthy newborns that died preventable deaths.

In some sense that's not interesting though, right? A newborn that never made it also had very few resources poured into them. From a purely utilitarian perspective, the worst age to die is 20-22, when you've got an education and a long history of people devoting time to making you a productive member of society, but you've had no space to contribute anything back yet.

I think there's a straightforward counter to that:

> From a purely utilitarian perspective, the worst age to die is 20-22,

that's easily adjustable: if 95% of newborns become healthy 20 yr olds, then just multiply the number of newborns by 0.95. The harder part is estimating total number of years lost, but it doesn't sound particularly out of the ordinary for a research problem in statistics/epidemiology.

The (terrible) thought experiment that perhaps gets the parent's point across:

Which "costs" more, 100k 20-year-olds dying today, or 100k babies dying today? The 20-year-olds have already had a lot of time, money, and opportunity poured into them, while the babies have not.

Compare it to the world population. It's currently about 7.8 billion. So if 50 million years are lost in total, everyone lost on average 50 million years / 7.8 billion ~= 2.4 days of life.

This way of looking at human suffering might be counter productive:

“oh, those slaves in [country] only cost me 0.03$/year, no big deal.”

Slavery costs me money and that's why it's bad? I don't understand.

I don't think that "total number of years of human life lost" is a good general-purpose measure. Specialists could perhaps apply that measure to the deaths of old people (people over retirement age?) but it would be crazy, in my opinion, to apply it to the death of a baby or a teenager.

As the hypothetical trolley controller, how many 50-year-olds would you kill to save one 16-year-old? And would you kill the 16-year-old to save a one-week-old baby?

I thought regulatory bodies considered age when considering regulations and the value of statistical life. Hence why regulations regarding kids toys / child safety products are so much stronger than say a toy for adults. Quick google search doesn't confirm or refute this, merely that the average VSL in america is 10 million.

https://en.wikipedia.org/wiki/Value_of_life

This does have this tidbit:

"Historically, children were valued little monetarily, but changes in cultural norms have resulted in a substantial increase as evinced by trends in damage compensation from wrongful death lawsuits.[38]"

> As the hypothetical trolley controller, how many 50-year-olds would you kill to save one 16-year-old? And would you kill the 16-year-old to save a one-week-old baby?

Organ donation prioritization does this analysis all the time. They don't publish the exact rules they use for matching but the most important are urgency, compatibility, locality, and survival benefit. Survival benefit is almost literally expected quality life years (QALY) from an organ transplant. A 98-year old will not get an organ that could give a 50-year old another 20 years of life, but a 20-year old that can survive for another year will not get an organ that the 50-year old needs this week. Compatibility and locality are the biggest problems apart from basic availability (too few organ donors, too many patients needing transplants). Once we can print/grow organs the calculus will reduce to prioritization on urgency and then QALYs if supply is still constrained in some way.

Economists and public policy wonks should do this for sure, as we make sense of the costs and benefits of certain policies. I imagine there will be 10 years of papers to come from this, and no agreement at all on what policies were worth it.

Change in quality adjusted life years, or variants thereof, are one standard for health-economic evaluation but are obviously very hard to get quality data on.

That would capture not only years of life lost but also the quality of those years. Long-covid seems to be a serious issue for a lot of people and an accurate estimate of that impact would be very valuable.

This article tries to do this:

“1.5 million potential years of life lost to COVID-19 in the UK, with each life cut short by 10 years on average”

https://www.health.org.uk/news-and-comment/news/1.5-million-...

If you're going to break things down to that extent you'd probably want to use quality adjusted life years.

Isn't this based on a rather questionable assumption that all excess deaths in this time period were due to undiagnosed COVID ? There's been a large spike in drug overdoses, and some increases in suicides that have been seen. Would it not be reasonable to assume that a number of these deaths were caused by lockdowns, and various other heavy-handed measures authorities used to try to control outcomes ?

All the data on suicides I have seen so far shows that there are no significant changes to the suicide rates outside existing trends.

For your other point about excess mortality being caused by lockdowns, you can view charts of excess mortality here: https://www.euromomo.eu/graphs-and-maps/

You can see that some countries, e.g. Norway had lockdowns but did not incur any severe excess mortality - but note that this only proves that you can have lockdowns with no excess, not that every lockdown is equal.

"Suicide attempt admissions have increased by 100 per cent on average during the pandemic," the release says. "Admissions for substance-use disorders have increased by 200 per cent."

https://ottawa.ctvnews.ca/cheo-joins-other-children-s-hospit...

While that's not the same as saying actual deaths from suicide have increased, there is a serious problem, at least here in Canada.

edit: Attempts up 90% in Colorado as well:

https://coloradosun.com/2021/05/25/mental-health-emergency-c...

Sure, I was aware of the increase in ideation and attempts but we were talking about excess deaths. I don’t like it when articles use percentages and omit the absolute numbers which are also important to give a sense of scale.

Suicide is a serious issue and shouldn’t be ignored. We should also apply the same logic to suicides as people are applying to covid - would these people have tried suicide anyway but covid/lockdown just brought it forward? Will we therefore have a drop in attempts now to mirror the spike up?

You're cherry picking numbers.

While a children's hospital in Colorado might have reported a 90% increase, across the whole of the US suicides were down 5%.

As far as I know, most countries are reporting an overall drop in suicides during the pandemic.

Suicides numbers are nowhere near high enough to make a significant difference, they'd have to increase by 1000%. And apparently they fell by 5% in the US in 2020 anyway.

And US drug overdose went from 72k in 2019 to 81k in 2020. So less than 2% of the economist's 500k[1] might be attributable to that.

You might point at road deaths, which have increased quite strangely, but again, wrong order of magnitude, by 2 orders even, increase of 4k compared to 500k.

So to answer your question, no, it's not a questionable assumption.

Perhaps you might consider the simplest explanation, a deadly virus, is the best one?

[1] https://www.economist.com/graphic-detail/2021/04/05/deaths-i...

Forgive the naive question: but is code like this typical for "data science"? It seems more like something out of a masters project... Huge script files with zero testing?

Theres a huge amount of data reshaping, mutating, and general wrangling going on here, how can one be confident without even a known input->output integration type test?

In my experience, yes. Data science is highly creative/improvisational and the resulting artifacts of work reflect that. FWIW, normal science is pretty similar. The clear step 1, step 2, ..., you see in scientific publications is effectively retcon done for the benefit of the reader's understanding.

Yes, sadly.

Testing is hard. The researchers who write this code typically don't have the necessary hands-on experience to write good tests, even if they had enough time in a day/week to actually do it.

Edit: Also the code tends to be very "high level", chaining lots of high-level API functions together, and even coming up with assertions to test tends to be a bit of a challenge. Testing such code turns out to be surprisingly difficult; you might end up just rewriting big chunks of your code in the test suite.

In my data science work, I've focused on writing tests for the complicated sections (e.g. lower-level string processing routines) and just trying to focus on the keep the other stuff very clean and readable.

What would "good tests" look like for something like this?

Generate synthetic data -> run model -> check expected outputs. Yes it a lot of work, but you're reaching millions of people with this model and correctness is paramount!

Similarly, even such simple test harnesses help when yourself or other go to modify the code. Having flags for "Did I break something" is very important.

The deliverable here isn't the code, it's the output. If "check expected outputs" is a subjective step anyway, why not just check that the output of the code when run on the data is as expected and skip the test altogether?

> If "check expected outputs" is a subjective step anyway

It isn't. It is the evidence that calculations in the code were done correctly.

It is like the problems in high school math class. The teacher controls the inputs so that the outputs are known. The teacher then runs the test problem past the student (aka the code). If the known correct answer isn't generated then we know the student didn't do it right.

How do you tell that the calculations were done correctly? Presumably you have some way of doing that.

Then, why don't you apply that way to the output? You don't need tests if you only need to do it once.

Checking that the code outputs what you think it does is a pretty low bar to clear. The fact that you can think of higher bars doesn't invalidate the value of checking this.

>How do you tell that the calculations were done correctly? Presumably you have some way of doing that.

Isn't that what we are discussing, how to tell if a piece of code works? You could do a formal proof that the code works (which is long and tedious) or you can test it (less long and tedious but less rigorous).

>You don't need tests if you only need to do it once.

It doesn't matter how many times you plan on running it. What counts is how much we value the output. I wouldn't bet $20 that untested code works correctly.

Synthetics only take you so far though. Real data almost always has unusual data (outliers), or the data will violate various assumptions that you've made along the way. In many cases there is no way to know ahead of time how the data will break your model. Which is why monitoring for model drift is so important.

That's part of the problem.

In general (and in my opinion), a "good test" is one that asserts that an invariant is always so, or that a property that is expected to hold under certain conditions does indeed hold under those conditions. Defining such invariants and properties for "data science code" tends to be difficult, and even when you define them it might be difficult or impossible to test them in a straightforward fashion.

And that's even before you get to the probabilistic stuff.

In general I think data science code is more "research-oriented", meaning that it is not run continuously like a software app and its requirements often evolve as more information is learned from the data. Research code produces results maybe once or twice a day (manually triggered) while software apps potentially service hundreds of thousands of requests a day.

Research code, because its requirements are not fixed and it is not frequently run, doesn't need to be as "stable" as a bonafide app. For a bonafide app, requirements do not change as often and the app is run virtually 24/7.

Once the research code becomes "productionized" however - i.e. it is deployed in an online system where uptime & accuracy matter - then I think absolutely it becomes more engineering-heavy and looks quite a bit less like this code.

Would be curious to hear others' thoughts on this distinction between research vs. production code however.

Many of the "good practices" in software development are primarily meant to reduce the cognitive effort on the part of the person reading the code.

I suspect it will be a downvotably unpopular opinion, but a typical researcher has a significantly larger cognitive capacity than a typical software developer. Consequently, a piece of code of certain complexity will look simpler, and be easier to manipulate for a researcher than for an average software developer.

It's always easy if you're the only one writing/maintaining the code.

Also, if your job is writing code, I'd bet you'd have less difficulty manipulating spaghetti code.

Huh? A researcher has greater cognitive capacity? That's a funny take.

My impression was ->

Software developers main focus is developing software so they spend a HUGE amount of time developing software, maintaining software, noticing patterns and bugs and pitfalls in software and thus they get pretty decent at writing software.

Data scientists main focus is developing models, so they spend a HUGE amount of time developing models, tweaking models, finding data, and cleaning data, and write basic software to achieve some of those goals.

I wouldn't expect Albert Einstein, Mozart, or Beyoncé to be some fantastic software developer just cause they're smart individuals. I'd expect people who spend a lot of time writing software to generally be the ones who write decent software.

This is a valid point, but I think in part the data science / machine learning cycle doesn't reward careful development like software engineering does.

Most of the time you're just testing the viability of things and just want to fail fast if they don't work. Getting too attached to some pipeline or abstractions is usually a bad idea.

I don't think this excuses bad code, but I kinda get it when it happens, since you can't just use best practices from the beginning like in software development

As a researcher, this is flattering, but probably wrong. Reading other people's code always sucks, but reading somebody's solo project hero code that doesn't follow any typical design patterns? That's nightmarish.

I'd argue it's more about startup time, or whatever the technical term is for the time between when you first start reading code and when you've built enough of the model in your working memory to be able to touch it without breaking everything.

That heroic developer doesn't need to concern themselves with other people's experiences, and they can just write whatever weird idioms suit their brain.

I think most research code bases are used by a significantly smaller number of people. Hence it's easier for the researcher to know what's going on in their code because they probably wrote a significant portion of the code they use and hence understand the assumptions, and limitations that are baked into the code. Whereas if you try to use a library normally you have to kind of figure those things out on your own.

That said, having used a lot of research oriented software packages over the years, researchers typically do not produce high quality software. Their objective is to produce papers, not software that other people can use. If someone else can use it too, then great, but that's not the primary motivation.

The purpose of testing in my example is for correctness, not continuous integration. I would argue regardless of "production" use, the fact that this is publishing a widely read newspaper makes it highly impactful and more likely to influence policy and decisionmaking, thus we want to know if its correct !

Well... that's partially why it's open-sourced. What fuller transparency of correctness is there an open-sourcing the entire analysis?

This code is above average by the standards of academia I've dealt with. You've got comments announcing what each part of the code is doing, and there is only one copy of each script. Normally, you would expect to have multiple nested directories with various variations of the same code, plus multiple variations of each script with cryptic suffixes.

Yes, data scientists frequently are working alone and don't care about readability so much as results. Also the code isn't going into production so they don't care about optimizing it by and large.

Sure, but physPop's concerns still stands - how can one be confident about the results? Is it just eyeballing "this looks right"? If so, how many anomalies are missed due to handwaving, and how many results are inaccurate?

I'm genuinely curious because I understand the need to move fast but is accuracy a necessary sacrifice? (or is there a trick I don't know about)

There's definitely a greater risk of a bug leading to misleading results. There's no real unique trick other than possibly someone else trying to replicate the results and catching an error, or trying to use the code on another data set and catching a mistake.

If I understand correctly, tests are for what the code should do. There is some business logic and you are asserting that the business logic does what it should do (the test). If you manipulate the business logic, and it no longer does what the test says the code should do, the test fails.

Here it is not as clear what the code should do. What should the amount of excess deaths be? In what ways would we change the logic such that the test case would break? If the input data set is static, isn't it more like a mock anyway?

I think for this reason you often see more sanity checks in research code because the should case is not as clearly defined.

>What should the amount of excess deaths be?

With several fake known inputs and there associated outputs we should be able to determine if the calculation is right.

The result on the real world data is not known but when calculating a statistic you should be able to figure out if you are calculating the right statistic or returning 42 for all inputs.

That's why it's a good thing that the code is open. The usual standard is the same type of code but without public access. This is definitely a step in the right direction.

As other comments indicate, how could you be sure the tests are testing the right thing? It's easy enough to slap on a few assert statements (or stopifnot statements, since it's R) about number of rows or something, but that's no replacement for manual inspection of the data. A code-based test can not be invented that will ever substitute for looking at the data in the raw, plotting it, and verification using your full mental capacities.

The only way to make sure it's right is the same way you'd do full verification of other code, going through it line by line and making sure it's doing the right thing. Tests can not do that, they can only assert some forms of intent, and they are not good at catching the types of errors that result from ingesting varied data from lots of sources and getting it into modelable form. It's ETL plus a bunch of other stuff going on here.

I didn't make the claim that a code-based test can be invented that will substitute for looking at the data in the raw, plotting it, and verification using your full mental capacities.

However, "the only way to make sure it's right is to ... go through it line by line" is a bold claim. There are multiple named functions and some unnamed functions in this example that could be verified for programmer mistakes (such as typing 1000 vs 10000) and edge case handling (edge cases that often arise from messy ingested data).

But even if they were tested, I'll concede that mistakes can still be made.

As already pointed out, this isn't an engineering product. They've released the source, did some sanity checks, and moved on with their busy lives. If you find an error - cool, go ahead and report it.

By sanity check you mean shows the numbers I wanted to see? Not saying there's necessarily something wrong in this particular code but I know first hands how little effort is put into validating the scientific codes and how much of produces just random crap.

Which is why releasing their code is critical. They lay it all out for you, with their pride/reputation on the line if it produces 'just random crap'. Writing exhaustive unit tests is probably not something they are good at, nor is it a requirement for them, nor is it the best use of their time.

I too know first hand that students and even academics who aren't properly trained and given the right tooling will write prototype code that's not production ready. Occasionally there's even a bona fide mistake, though usually it's just very un-generalizable. It's part of my job to make some of it more production ready. That's the best use of my time and training. Conversely, I'm shit at generating useful research ideas or writing papers. Just don't have the right combination of intuition/training/experience. Good thing my academic institution has the resources to employ them and me.

How can you know which numbers you want to see?

With tests it is often best practice to write the tests before writing the code that gets tested. But what if you don't know ahead of time, and can't possibly ever know ahead of time?

Writing "tests" for this would be about adding stuff afterwards, like "the max and min values of this column were X and Y". But that is expected to break, if anything changes, because it's not testing anything useful.

My question is: what is one concrete test here that would be useful and actually provide confidence that the code is doing what it should be doing, and how does that test provide better sanity checking than inspecting the tables at each step of the process?

For something like this, the code is more an aid to analysis rather than the product itself.

in R particularly, there is a wide range of code quality, by developer standards, yes

It should be said that the statistics themselves do make for some 'guide rails' .. the stats are quite demanding, require a lot of upper-division training to use correctly, and visual feedback can course-correct as the results are iteratively found.

As said in other comments, the content is often associated with a research goal that is weighted more than code-quality. In contrast, a general purpose development language has very broad application, and probably can go wrong in very broad ways. The coder is getting feedback on results, but often lots of code review and expectation of professional results.

I picked out an R ETL sequence from an article last year, and I still pull it out once in a while, to see the extensive, clever and (to my eye) really hard to read R data ingestion and manipulation. Personally I think it is a fair tradeoff to say that the expertise to use this environment is a bit of a filter, and the rigor that the results (therefore intermediate results) demand, does move the expectations on code quality ..

As for tests, there is probably no defense on not having tests also.. it is probably going to be more common as the field of R and data analysis inevitably grows.

Yes, you can see the divide in the popularity of interactive notebooks like jupyter and observable. To programmers who use a standard git-based workflow, the interactive notebooks are anathema because you can't use version control to compare code state and monitor progress, and because of the global variables. But to many quantitative researchers for whom software version control isn't a large part of their professional worldview, it seems normal to just keep adding to the same document until you hit a state that seems to work, without much emphasis on checkpointing and navigation between checkpoints, and testing at each checkpoint.

This is pretty good by data-science (and other research) standards.

That is quite common for R language, it is a scripting language that you are suppose to explore as you use. So if you want to test it, it's very simple ... Just run the script until the part you are interested and then plot the hell out of it.

It's widely known on the "theory of testing" that data oriented systems can't be tested without immense amount of effort. If you are interested there are plenty of academic talks about the subject.

Would love some commentary on why this is an unpopular question.

Good software practices don't only have to be for "production" or CI or large teams of coders. Testing for correctness could be seen as part of the work of delivering a high quality product/graph/prediction/model.

Reply on news.ycombinator.com