On “Armchair Epidemiology”
scottaaronson.com
scottaaronson.com
(1) It treats authorities and non-authorities as monolithic. Far from it. There were plenty of serious, well qualified epidemiologists who were warning of the danger, and they were right. There were even more non-serious people using their popularity in completely unrelated fields to minimize the danger, and they were wrong. Sadly there still are, most definitely including here. Yes, the people in the actual seats of US federal power got it wrong, and plenty of amateurs got it right, but it's more complicated than the "don't trust experts" narrative.
(2) Sample of one. Even if it weren't for the above, and the two groups were monolithic, this is just one case where the authorities were wrong and others were right. Yes, it's a very prominent case because of the consequences, but prominence is not the same as statistical significance. Treating it so is a form of cognitive bias/error.
The real lesson here is not to distrust experts or authorities. It's not to trust conspiracy theorists or people with whom we empathize more because they're in our own industry. It's to trust science itself - empirical facts and analysis applied to facts. Both experts and others can help us to interpret those facts, put them in perspective, correct common errors in analysis, but ultimately it should be the facts that rule. The reason so many authorities and non-authorities were wrong is because they rejected the scientific approach, not because of who they are.
First of all, I think calling this a single event is reductionist. We're talking about weeks- months, even- of experts and non experts warning about this, and the systems of authorities failed completely to react in time. We're not talking about failing to heed an important warning once, we're talking about a long string of choices here.
Secondly, This isn't just a prominent case, it is a very important case. Treating this as a series of bernoulli trials with a single (statistically irrelevant) failure is an extremely reductionist model of the reality here.
While your conclusion of trusting science, empirical facts and analysis rings true to me, I think we should be wary of overly simplifying the subject matter.
It seems like an Epidemiologist is most likely to be a doctor with some knowledge of statistics. On the contrary, I would have thought that a statistician/operations research person with domain knowledge of medicine would be a lot better suited to study epidemiology.
A lot of the existing models also seem really naïve for something an entire field depends on. Tradeoffs between human life lost due to the disease and effective human lives lost due to economic downturns seem entirely unstudied.
Czechia's actions of forcing people to using masks seems to have worked amazingly well for flattening out the curve. The "science" not seems to support it, and official recommendations are now moving back to recommending masks for all. The popular hypothesis for this, is that it stops nonsymptomatic carriers from being spreaders.
Now this sounds like a fairly obvious thing to realize, and to an extent, existing models should have caught this a lot earlier.
I grew up as a kid with an almost religious belief in Science. Over time, I have come to realize that a lot of our sciences are at best empirical and at worst entirely unverifiable. I am a 100% certain, that in a 100 years, we will look at medicine and nutrition of the 2000s, the way we look at humors and blood letting.
https://www.reddit.com/r/Physics/comments/frsd16/the_best_th...
If you look at the stuff coming out of Imperial and Oxford in recent weeks, there are a huge number of basic problems. Their papers can't be replicated for multiple reasons. They're using known-bad input data. They're not providing uncertainty bounds. There's no peer review. The Imperial paper assumed constant hospital capacity. Their papers reach diametrically opposite conclusions. They have a track record of model failure.
It's all practically a poster child for the replication crisis.
The /r/physics post criticises physicists who write some code to do basic curve fitting. Has he looked at the Oxford epidemiology paper? It is by all accounts literally curve fitting to the first 15 days of outbreak and reaches diametrically opposite conclusions to the Imperial paper.
It criticises people who write papers and then upload them to arXiv because reporters will find it and create panic. Would this person prefer people to use the Imperial/Oxford technique of sending papers directly to journalists, and entirely skipping the whole upload to arXiv step?
It blames physicists for creating "denial": "You become a punchline to a denier that says, they can't decide if there's going to be hundred thousand cases or a hundred million cases! Scientists don't know anything!" - guess what, they're saying that already and are going to say it a lot more because epidemiologists themselves keep contradicting and criticising each other for doing bad work, in public. Additionally, there are enough people with scientific literacy to understand the limits of statistics and modelling out there. They aren't as easily confused as this person rather seems to assume (does he have any data showing that people conflate epidemiology with physics?).
Right now I absolutely want to see papers written by physicists studying COVID-19. Why - because physics is a significantly more rigorous field than epidemiology. I trust the average physicist to have at least slightly higher standards, for instance I trust them to at least pretend to care about statistical uncertainty, and I suspect many of them will upload their source code. I don't expect any of them to email their paper straight to known-friendly newspapers.
The whole thing has me desperate to ignore everyone and read a book. What a joke; to think we are all somehow experts.
When you don't have relationships with a technical community, there's no interesting anchor to stop you from gyrating with the news of the day.
It seems like "back in the day" people trusted the experts, becuase well, there weren't many alternatives (?); but now thanks to the internet everyone can google for a while and feel like they have a worthwhile opinion.
It seems like now it's possible to find an expert (and/or data) to justify any opinion.
I am not saying this is bad, but that it's just different. The game has changed.
Because there will be. Of 'something' but we don't know what that will be.
Seems to me then what you want to a robust set of responses that do not require people to make decisions based on god knows what. Because in the future you don't know ahead of time what's going to be helpful and what won't be.
What you want to public to do is reflexively anything that might help. And then later tell them what measures they can back off on. Not the other way around.
What's so hard about masks? That's one area that experts seem pretty consistent on, and what they say makes sense. It is slightly complicated but not contradictory.
Wearing a mask doesn't do that much to prevent you from getting sick. But it does a lot to prevent you from getting others sick.
If there is a limited supply of masks, the best people to give them to are people are at high likelihood of being sick and come face to face with lots of people who get sick easily. Which is to say medical staff.
If there were masks to go around, we'd want everyone wearing them as a precaution. So that you're not getting people sick even if you don't yet know that you are sick.
Some countries, like Taiwan, already make enough masks that they can actually do that. The USA does not. In time we should fix that.
Can you find anyone whose expertise is generally respected who says something that disagrees with what I just said?
Eg, everybody, everywhere said that NodeJS has amazing performance for high io workloads. If you look at benchmarks however, the nodejs systems cannot keep up with older tech, so now the argument is “we use it because it’s a nice environment, not for performance.”
I am trying to focus on textbooks now, but with so much self publishing I think that it is only a temporary strategy
The second problem is that we're clearly terrible at collectively digesting new information uniformly. New information is evolving our understanding and things said a week or months ago no longer reflect our current understanding. But not everyone's subscribed to the same newsletter, so we're not on the same page. Back in January, based on SARS and MERS, and how much that affected the western world, the comparison to the flu was apt. Was. It's not longer January and that comparison is bad and is liable to get you branded a right-wing capitalism-death-cultist.
There's no way to read only what's changed, so either to get into the muck and drink from the firehose, or defer to experts, some of whom are actually paid to deal with this.
Go read a book, the muck'll be here waiting for you when you get back.
When you do, ask the experts, when you get back and update your knowledge correspondingly.
Scientific experts across the world have been alarmed for months now. The only problem is that, as with any new and uncertainty situation, the error bars started absolutely enormous, so the range of expert opinion has been wide. This is the case in any crisis. With the benefit of hindsight, you can always find an expert who was wrong, but that doesn't mean anything.
It's important to say this because later, you're going to start to hear bloviating from politicians who failed to act for months. "Nobody knew this was coming." "Everything that went wrong was solely the fault of $enemy_country or $rival_party."
A standard of "scientists dabbling in journalism to improve communication" instead of "journalists dabbling in communicating science" would likely have gone a long way to improving the civilian response and trust in experts.
There's also the factor of the "Noble Lie" dishonesty around wearing masks. I suspect intellectual-ish contrarians (like Aaronson et al) are more angered by this than the average person, but we ended up with shortages and sellouts anyways: a significant number of non-experts didn't buy into that noble lie so the experts/organizations that pushed it (not to be confused with ALL experts) burned a lot of good will to basically no effect.
ETA: I considered replying to your comment about the Harvard epidemiologists but would rather edit it in here to avoid two replies to one person: I think that's a great example of my point. The story wasn't broken by bloggers, but bloggers were more likely to be amplifying the concerned experts than our "traditional media powerhouses."
> “Many epidemiologists and people who are following this outbreak closely are assuming that it’s probably quite a bit more widespread than the case counts suggest,” said Michael Mina, assistant professor of epidemiology at the Harvard T.H. Chan School of Public Health. “Many people also think that there’s probably over 100,000 in reality out in mainland China, for example. We just don’t capture all of those through the case reporting we have. … We can assume that this is growing at somewhat of an exponential rate, and it will continue increasing in scale.”
This story wasn't broken by bloggers!
Source: https://news.harvard.edu/gazette/story/2020/02/as-confirmed-...
I also trusted in our institutions. I knew what lengths that they went to in SARS. I understood how much thought has gone into public health. I found articles explaining why one would rationally ration available tests. Even when it was clear to me that we were in for a pandemic and millions would die, I completely failed to predict that we'd be willing to put Western countries in lockdown - or that lockdown would be as effective as it has been.
I probably came to the right conclusion before Scott did. But still the only thing that I did right with my knowledge was to tell my wife to buy a put on the DOW back when it was still rising.
we are being forced to adapt and to develop new ways to relate to information, to asses it's trustworthiness, etc...
it's an epistemic phenomenon. perhaps another facet of this phenomenon are the filtered information bubbles.
But this pandemic really puts the pressure on us to face the reality of our current information society.
> This was notable to me for being the diametric opposite of the actual lesson of the past two months.
So, because some (unspecified, unnamed) experts weren't able to predict the future with perfect accuracy, we should give equal credence to "random people" instead?
Medical professionals, scientists, and public health experts are still the best available source of knowledge about preventing the growth of this pandemic.
Spreading the message that we should ignore epidemiologists is wrong at best and dangerous at worst.
That wasn't my takeaway.
My takeaway is that if you have some intuitive sense that shit's about to hit the fan and authorities keep telling you to stay calm and don't take any special precautions, all while danger is clearly growing, it's best to lean towards following your intuitions gradually just in case things ramp up. You shouldn't buy 10000 rolls of toilet paper, but all those people who thought, "Hmmm, maybe I should buy some masks and a couple extra bottles of hand sanitizer. Maybe not visit the movie theater anytime soon as well" while authorities were saying "Go to work and be normal. Masks are useless"... well, they were right and it would've been better to take those then "unnecessary" precautions.
Reminds me of people talking about government surveillance pre and post Snowden. If you wanted to encrypt all your messages and avoid Facebook, etc, you were a conspiracy nut because it was impossible to surveil everything. Overnight it became "Of course they monitor everything. Why wouldn't they? But we have end-to-end encryption now and we can completely trust it."
The experts are definitely what we need, but not all experts are created equal. It’s like during wwi - it took a while to weed out the useless commanders/generals since the peacetime systems failed to promote based on ability
(Edit: Incidentally, I think we have some great people in Alberta at the moment. Very lucky for us)
His default was to listen to those in authority and responsible. Instead he wishes that he listened to those who did independent thinking and their own projections with verifiable reasoning from the data.
This doesn't mean don't listen to epidemiologists. There were plenty of epidemiologists who were plenty concerned in early February. It means do not pay particular heed to epidemiologists who were selected to be listened to via a politically motivated process.
And my own editorial. Go read the book Superforecasting. The people that Scott is recommending listening to are exactly the kind of people who do better at making short term forecasts than the majority of recognized experts. And the reason why their forecasts are better is exactly the same as why they got it right this time.
Quantitative thinking is something we as a society are still learning to trust. We've come to trust it in baseball. (Go read Moneyball.) We've come to respect it in politics. (Nate Silver established a following for a reason.) But we have yet to internalize as second nature the idea that we should sanity check everything with people who clearly think in a quantitative way. And that we should attempt that kind of thought ourselves.
In the long run I believe we will. And perhaps reflecting on who got COVID right and who got it wrong will help us get there sooner.
I think I agree with btilly. A lot of people are upset right now by the lack of traditional deference to experts, by which they mean academics, not e.g. actuaries working for large reinsurers or even actual working doctors in many cases. In contrast I find it exciting.
What we're witnessing here is a total and complete democratisation of data and analysis. An entire planet's worth of brainpower is focused on this subject right now. Every day those people are using the internet to quickly sift through data, statistics, reporting and analysis to try and make sense of the chaotic picture with which we're presented. Some are trying to establish the bounds on what we know about the infection, others are looking at how to rapidly scale ventilator manufacture and so on.
If you've followed the replication crisis closely over the years, like I have, one of the overriding themes is how academic analysis is made brittle by:
1. The subtlety and trickiness of statistics.
2. The vast extent to which it's relied on despite this difficulty.
3. The relative lack of cross-discipline collaboration that could solve the combination of 1+2.
Lots of fields (not restricted to psychology) have been seriously battered by the collapse of whole research areas. I've heard that one reason VCs prefer to invest in software startups over biotech is the typical biotech startup begins with an academic paper, and around 50% of the time it doesn't replicate. Even AI has had replication issues!
One of the conclusions that's been drawn is that many academics don't have enough statistical training to do what they're trying to do, partly because statistics is genuinely extremely hard. In computer science the general maxim that you don't invent your own ciphers because they'll probably break is well known - everyone relies on standard cryptography made by a relatively small community of people these days. Advanced statistics feels about that level of difficulty to me, based on what I've seen, but everyone rolls their own stats. Perhaps there aren't enough statisticians to go round, but there's also a cultural issue. For instance academic modellers rarely work with professional programmers to productionise their models, a practice that's standard in the private sector.
So yeah, btilly has it right. We're seeing a slow, slow, veeerry slow recognition that truly quant thinking is so rare and highly valuable that it's more important than the background of the person. Finance already went through this process years ago: the clear-out of traditional finance types with traditional finance culture in favour of quants doing mathematical analysis is large done already. In other areas of the economy it's barely got started. In the UK Cummings did at least try to hire a superforecaster into Number 10, to his credit, but the media immediately went bezerk and engaged in a massive smear campaign. The guy just walked away in disgust. A superforecaster in government sure would be useful right about now :(
I’m sure someone will be along momentarily to tell us that it’s no worse than seasonal flu.
People are pretending this is all WHO's fault because they said some wrong stuff in early January, when the correct conclusions were not at all obvious. And they've been completely correct and transparent since late January. It's almost April now. At this point, it's just a worn out excuse.
There's also some evidence that widespread usage of masks may be effective in lowering the risk for such a disease, but deploying widespread usage of masks equitably on a global scale is nigh-impossible right now, so it's counterproductive to focus on it. Distancing is achievable instead, and leads to a similar result.
You don't need high grade medical masks to significantly reduce the amount of transmission. Any sort of fabric-based covering on your face can help, just make sure you can comfortably breathe through it. Fabric is cheap and ubiquitous, it's literally everywhere.
Oh look, the results are already in! https://news.ycombinator.com/item?id=22727761
This article: https://virologydownunder.com/past-time-to-tell-the-public-i..., plus the rising mortality numbers, spurred me to start warning the medically vulnerable people in my life on Feb 23. I am very grateful to the authors for helping me do the right thing.
At this point, it seems to me that our experts are still letting us down in two important ways:
1. They aren't rebutting this implicit notion that any mortality risk below 1% can be rounded down to zero. As a consequence, the idea that it might be a good idea to let younger, healthy people just go out and get the virus and get immune seems to still have currency, and this is really dangerous. We can't open things back up until we have a credible plan for driving the infection rate toward zero. Leaving aside the risk of life-changing injury, a mortality risk of eg 0.5% applied to eg half of young adults would still be by far the most dangerous thing that happens to most of them this year, and a huge personal tragedy for many of us. Do the math and work out how many people you know would die.
2. Simple, inexpensive measures like widespread masking, handwashing, gloves and eye protection will clearly be important components of any credible plan for driving the infection rate toward zero, unless we get an early vaccine. The experts should be hammering the point that we all need to be fastidious about those things whenever we open back up again.
The fact is, that many people's internal information filtering algorithm is lacking, small preparation is sufficient 90% of the time as long as you are looking at slow moving historic signals, it's about small risk mitigations early.
As played out as it is, this is further proof that the medium is the message, people engaging in unidirectional or bidirectional firehose messaging (cable tv, twitter, facebook) have tended towards panic or dismissiveness, people following historical information and slowly changing documents and global forums have been able to take prudent low cost risk mitigation, and by doing it early have operated comfortably on a different wavelength than people panic buying.
Unstructured messaging and it's cognitive overhead is a problem, unstructured messaging means that even if the inbound message is about the issue currently in your head, the design of "alerts" trigger a disruptive near context switch, like a digital doorway effect that sabotages deep thinking, deep thinking is required for prudence.
People still don't understand how bad this is though, for Americans, I'm only just now seeing a few people and articles connect the dots on bio warfare preparedness and how our asymmetrically weak and discordant response puts the U.S. at increased risk from a national security and power projection perspective. These aren't truths per se, just probability curves.
As an example, since mid December I've been gently decreasing my downside risk, divesting and putting a few dollars here and there into things like masks and non perishables, already had a bidet, water filter, UV lamps, etc. No special preparation just things I already have around the house for my aquarium, camping, spray painting, hot ones challenges, etc. I haven't had to wait in any grocery lines or experience any personal discomfort or anxiety, and my slightly early social distancing was subtle enough most people didn't even notice. I was telling people who asked "These might be historic times, low cost preparation is in order."
All this is from an individuals perspective, businesses should already be operating with risk management plans.
This is the opposite sentiment to "there are no libertarians in pandemics." Keen perception of reality is unlikely to be concentrated at the top of political hierarchies. Competent Authority isn't an oxymoron but it's distressingly unreliable.
> For now, I think the risk from the ordinary flu is much much greater! But worth watching to see if it becomes a real pandemic.
Nope. Lost all credibility right there. Do not pass GO, do not collect 2 minutes of attention. Sorry.
Oh, you've learned something from the error and you've updated your priors? Great, good for you. You're still unreliable.
Plenty who were paying attention took it seriously
Are there gonna be any legal repercussions to these?
[1]https://www.wsj.com/articles/is-the-coronavirus-as-deadly-as...
[1]https://thehill.com/homenews/administration/490138-birx-says...
P.S. They even make the claim that COVID-19 has 1/10th the mortality of seasonal flu :facepalm:
That is a prediction. It is not yet a fact.
Right now we just don't have a good idea of how many people 1) how many people are currently infected, 2) how many people have already been infected and recovered without being tested, 3) how many people were already immune and would never have been infected, 4) how the way you are infected changes the outcome of the disease.
The mortality rate depends directly on those numbers. Until we know those numbers, we just don't know for sure. With H1N1 initial estimates of the fatality rate per infection were almost two orders of magnitude too high. A similar error may or may not be happening here.
The best we have right now is a tiny handful of limited, non-representative, "accidental experiments" like the Princess Diamond, that may or may not represent how the disease spreads in the community (note question #4). This will however soon change as more randomized testing is done, and in particular, anti-body testing to determine who already got the disease and recovered.
"We need more data".
No you dont need data, look at the ICUs you fool. They are struggling to find ventilators and in worse cases with lower capacity, even just for beds to deal with the influx of patients.
But you can wait for the data. They are incoming. Hopefully we will still be here in 2 months to discuss.
https://virologydownunder.com/past-time-to-tell-the-public-i...
A. Politicians have interests representing themselves as well as their country, and thus have given dishonest advice (masks don't help) with an ulterior motive (save them for hospital), downplayed the problem, and covered up the number of cases.
B. I think a lot of actually qualified experts (doctors, scientists) to think the public can't handle the truth perhaps? I still have yet to see anybody in any position of respect or authority simply say the bald truth: "A few million Americans will die, at least, unless some radical cure happens. Social distancing is a drop in the bucket, ventilators make a tiny difference. The vast majority of young people will be totally fine"
I'm disappointed in our society for failing to bubble up and synthesize good information quickly.
Sorry, but look at how China, South Korea, Taiwan etc. are doing right now - this doesn't jive at all with "social distancing is a drop in the bucket". Even Western countries/places which started social distancing early in their local outbreaks look like they'll be a lot better off than places which didn't.
> The vast majority of young people will be totally fine
This is also quite wrong, unfortunately. A sizeable minority of young people will need hospital treatment for COVID, and perhaps intensive care. Given overwhelmed healthcare systems, they won't be "totally fine" at all. Of course, older folks are at increased risk.
https://www.youtube.com/watch?v=gAk7aX5hksU
A few points of interest: (He is a very respectful fellow so apologies if the summary seems a bit to the point)
- Masks help and he is slightly puzzled why the west hasn't adopted that culture
- Their testing and tracing helps and they doubt the US would allow their level of privacy invasion
- Korea is doing well because they have had to respond to pandemics and were prepared (as are Taiwan, Singapore etc)
I'm taking China numbers with a LARGE grain of salt (basically I don't believe them at all given how they have lied from day one of this pandemic). I'd wager South Korea has a level of preparedness that few other countries in the world has, given that they share border with a hostile country.
Overall it seems that South Korea and Taiwan were very successful in restricting and tracking travelers (I read Taiwan totally closed the border for foreigners a couple of weeks ago) and were therefore quick to quarantine people coming from countries with outbreaks.
This is a frustrating and sad thing to read here. Social distancing WILL save lives. Even if it is a "drop in the bucket", and I don't believe that to be true, once hospitals are at capacity every SINGLE severe case that social distancing prevents or delays is potentially a life saved.
> ventilators make a tiny difference
Would you say that to the families who have a loved one on a ventilator right now?