Hacker News Reader

Comment by jaaron | Hacker News Reader

Views still need to be backed by solid data and in this case, the Stanford study the article is highlight has a number of issues with its data:

"I think the authors of the above-linked paper owe us all an apology. We wasted time and effort discussing this paper whose main selling point was some numbers that were essentially the product of a statistical error."

"I’m serious about the apology. Everyone makes mistakes. I don’t think they authors need to apologize just because they screwed up. I think they need to apologize because these were avoidable screw-ups. They’re the kind of screw-ups that happen if you want to leap out with an exciting finding and you don’t look too carefully at what you might have done wrong."

source: https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaw...

View on HN

I don't think that Ioannnidis needs to resign or something, but he should retract the paper and recuse himself from the ongoing debate about this issue. It was their group's decision to

a) Run op-eds in the WSJ and get on Fox News with their theories

b) Create a faulty study with biased data analysis (of all people, Ioannnidis should be very sensitive to this -- he's famous for his paper "Why Most Published Research Findings Are False."

c) Publish the flawed paper as a preprint without peer review.

d) Publish the results of the L.A. serological survey without even a preprint so that the methods could be looked over.

If you publish a study that has huge societal implications before peer review, then Twitter is going to be your peer review. Deal with it.

I don't blame them for posting a dodgy preprint (that's what preprints are for) but I do blame them for going on a press blitz for their dodgy preprint. It's not a responsible way to use their platform.

Well, some may find posting said preprint particularly questionable simply because it had Ioannidis' name on it, and he's primarily known for his emphasis on standards of evidence.

As somebody who was always annoyed by the weird amounts of odd attention that Ioannidis' "standards of evidence" papers got, I hope that this makes people reevaluate them.

The "standard" he set up was a straw man, and not how any practicing scientist I know actually looks at papers.

And when he allows his name to be used to push highly politci contrarian takes, all the weird attention his other papers received starts to make sense. It's not about the science, it's about being contrarian, at best, and at worst ways to subvert the public's view of science by deriving them on what it really means to publish a peer-reviewed paper.

In a video promoting his study, he said the WHO said 3.4% of people who get infected would die. Which seems like a blatant lie with a strong agenda: https://www.youtube.com/watch?v=cwPqmLoZA4s&t=12m54s

What they actually said was:

> Globally, about 3.4% of reported COVID-19 cases have died

He also says it doesn't seem to have a higher chance of killing you than seasonal flu for each person infected:

https://www.youtube.com/watch?v=cwPqmLoZA4s&t=1h9m50s

And says in there that Imperial college was dangerously wrong to use a 1% IFR estimate in their modeling. I believe this interview was near or after New York City's city fatality rate was already over 0.1%, yet he was still saying Imperial's estimate was an order of magnitude off.

Throughout the whole thing he cherry picked countries with recent extreme case growth and thorough testing to take advantage of the death lag and claim they point to a tiny CFR (and IFR). In each case it steadily rose after case growth slowed and lagged deaths continued, as expected.

In the Ioannidis serology study, one of the researchers's wives recruited people who wanted a test, saying it would let them know if they could get back to work (potentially biasing things for people who suspected they had had it).

In each case it steadily rose after case growth slowed and lagged deaths continued, as expected.

Where do you see that? The data I see says the fatality rate of all forms has been dropping over time, as expected from prior epidemics where this also occurred. It's the Imperial papers that have been discredited by now, as their projections were way off in all respects - if they were right Sweden would be a bloodbath at the moment.

I agree the serology survey was not a perfect study - more for statistical/confidence interval reasons than the researcher's wife. But it may not matter. The general conclusions of the serology survey are matching many other pieces of data and evidence from around the world. For example:

"The latest figures from Italy show (pp. 12/13) that 60 of almost 17,000 doctors and nurses who tested positive died. This results in a Covid19 lethality rate of less than 0.1% for those under 50, 0.27% for those aged 50 to 60, 1.4% for those aged 60 to 70, and 12.6% for those aged 70 to 80. Even these figures are likely too high, as these are deaths with and not necessarily from corona viruses, and as up to 80% of people remain asymptomatic and some may not have been tested. Overall, however, the values are in line with those from e.g. South Korea and give a lethality rate for the general population in the range of influenza."

From https://swprs.org/a-swiss-doctor-on-covid-19/ - the source document is in Italian but the data table is quite clear even to someone who doesn't speak it. In other words for Imperial's paper to have been reasonable they'd have to assume the entire population had the mortality profile of 60+ year olds, which is just wrong.

It’s not just that it was a faulty study.

It was a study that was about an ongoing crisis that has led to hundreds of thousands of dead people in a couple of months that had a contrarian result, had not been through peer review, had not been double or triple checked and was still promoted heavily through news media.

That it happened to also be faulty made it a tragedy. It’s farcical nature had been established earlier.

It was an opinion piece, not a study, and was labeled at the very top as such:

https://www.statnews.com/2020/03/17/a-fiasco-in-the-making-a...

Opinions don't need checking, peer review or double-blind controlled studies to be heard. That's why they're called "opinions".

"Opinions are like assholes: everybody's got one and they all stink!" - Unknown (NOT DeadPool; much older than that!)

I want to add one other piece of information to this discussion, because Ioannidis is somewhat central to the point the original article is trying to make, and it is ignored by the article (because it significantly weakens some of their arguments).

Had the study simply been released and criticized, then much of this current controversy would've been avoided, but that didn't happen. Instead, Ioannidis did the rounds at several conservative news outlets in the wake of the study's release, and promoted the study's findings as the new "best evidence." This aided in pushing certain social and political agendas. Many scientists (especially on Twitter) found this specific behavior to be unacceptable, especially from a scientist that they also felt should "know better."

Will Happer should "know better" too, but as Sinclair said: "It’s difficult to get a man to understand something when his salary depends on his not understanding it"

Did those people criticise other epidemiologists for doing the same? Immediately holding press conferences appears to be standard behaviour during this outbreak; for example, Imperial and Oxford both did the same. In fact in the Oxford case the press published stories about the paper before it was even available anywhere, let alone in a journal.

I understand why at the moment peer review is being short circuited by everyone, regardless of political or epidemiological position. The world is hungry for news and journalists will interpret papers with or without the help of the authors. Might as well have their assistance, in these circumstances.

Your remark about "conservative news outlets" shows what's really going on here, I suspect - just another day in politically biased academia-land. If you talk to anything other than CNN then you're bad and wrong. The same has been seen with Alex Berenson, a former NYT journalist turned novelist who has never really taken part in politics nor even really discussed it much in public. Once he started tweeting skeptically about the actions of governments and academics, all based on public data, he discovered only the conservative news outlets were willing to interview him. And then that mere fact alone was used by the left to attack him, although his points were all based on actual verifiable facts and had no pre-existing political bias.

Ioannidis made a lot of enemies by kickstarting the replication crisis. Now it looks like some see a chance to get back at him.

> Views still need to be backed by solid data and in this case, the Stanford study the article is highlight has a number of issues with its data.

That articles backed by solid data is certainly the desired state, but it is very far from the current state of affairs on COVID. There are many articles that tout opposing views that are pretty liberal with data. Requesting an official apology (which, in science is almost unheard of) is simply an attempt at harassment. Mistakes happen, you refute errors and move on.

Scientists must be allowed to explore and advocate non-traditional views. Otherwise (and very quickly if you let hecklers in) you create static beliefs that live on for a long time, whether true or not, because the bar for challenge is very high and an error ruins a career.

You've quoted the conclusions, but in the spirit of the STAT article, can you also quote the relevant scientific debate in your article, and what specifically leads to such bold conclusions (e.g. "owe an apology", "wasted time and effort", "statistical error", "screwed up", etc.)?

I skimmed through the article and, because I'm not a statistician, I could not follow what was so blatantly wrong, if we were to charitably assume that the two sets of authors may have different premises or interpretations.

Edit: In the spirit of the STAT article (edit: removed extraneous text), I searched for "Ioannidis Gelman" (the latter being the author of your piece) and found: https://www.mercurynews.com/2020/04/20/feud-over-stanford-co...

> In response, on Sunday, the Stanford study’s authors said they are planning to soon release a detailed appendix that addresses many of the “constructive comments and suggestions” the team has received.

> “This is exactly the way peer-review should work in scientific work, and we are looking forward to engaging with other scholars as we proceed in this important work,” said Dr. Jayanta Bhattacharya, professor of medicine at Stanford University, who along with colleague Dr. Eran Bendavid, also assisted with the USC study.

Edit2: I'm not sure if this is the appendix referred to (probably not because the file has an older date in it), but there is a statistical appendix in the cited article: https://www.medrxiv.org/content/medrxiv/suppl/2020/04/17/202...

> what specifically leads to such bold conclusions (e.g. "owe an apology", "wasted time and effort", "statistical error", "screwed up", etc.)?

They used a test with a false positive rate of about 2%, and found a positive rate of about 2%. It is impossible to use this information to establish a lower bound on the number infected. However, by screwing up the computation of the confidence interval, they claimed a huge lower bound.

Thanks. By the way, the STAT article links to https://undark.org/2020/04/24/john-ioannidis-covid-19-death-... in which Ioannidis is quoted:

> “We have been working very hard to address all the comments raised by many colleagues and I am always grateful to receive constructive criticism,” he said. “While we received thousands of congratulations for this work, I cherish far more the comments of the scientists who criticized our work and I thank them deeply for their comments.”

That's an admirable sentiment, Ionannidis must be such a wonderful guy.

However, the damage has already been done. He's already done his media blitz, so no matter what happens next, effectively zero people will hear the correction (or more likely, subsequent retraction). It'll just become yet another piece of zombie information, which will appear in dishonest talking points or gish gallops. The quality of public discourse and the reputation of science will continue its decline, and we'll be talking about the next example of this tomorrow.

If we try to be charitable, Ioannidis was too confident in his research team who made mistakes; for example:

> The authors said by email that they used a built-in Stata function and aren't sure themselves how the software used the input weights. I suspect they misapplied that function (too complicated to tweet why) but I don't know Stata well enough to be sure; it seems neither do they.

https://twitter.com/wfithian/status/1252692357788479488

Then, Ioannidis probably didn't double check the work in depth because of confirmation bias and over-confidence in his protégés.

I checked with a friend who has a PhD in statistics specifically about this Twitter thread (which was referenced in some of the articles about this): https://twitter.com/wfithian/status/1252692357788479488

One claim Will Fithian makes is:

> The errors are not debatable and can be seen in these two screenshots of the supplement: 0.0034, the standard error meant to measure uncertainty about prevalence pi, is not the square root of 0.039, and the variance of a binomial estimate of proportion depends on the sample size.

My statistician friend says:

> In fact the math checks out in favor of the authors. Square root of .039, then divide by square root of 3330. .= 0034 just like reported.

> So the error is debatable.

> Now, sometimes you should be using the standard deviation, not the standard error, but that isn't his claim. The .0034 is absolutely the correct standard error, unless there is a difference in terminology for the field or something.

There was also an absurd amount of selection bias in their population. They recruited people for a COVID-19 antibody screening, and there were emails encouraging people who suspected they had recovered from the disease to get screened. If the researchers knew about the email, it's actionable research misconduct.

If they didn't know (a coauthor's wife sent the email), then it's just very very bad work.

However what about the USC and the NYC studies? Are they wrong too?

The NYC studies actually look halfway decent. The big problem with serological tests is that even good ones may have, say, a 2% false positive rate. So if you test a population where nobody is infected, the test will say that 2% have been infected.

To get good numbers, you need a population where lots of people have been infected. That will allow true positives to swamp the false positives, which is what happened in the NYC study that claimed 21% of people had already been infected. Even if it's really only 19%, that doesn't change much.

You also need a good random sampling procedure. IIRC, the NYC study tested people shopping for essentials at mid-day. This underrepresents essential workers, people who are really good at self-isolation, and people who are currently sick. But still, it's not too bad.

The New York City numbers suggest that very roughly 1% of people infected with the virus will die, depending on how you count probable but unconfirmed COVID deaths reported by NYC. This could be off by a factor of two in either direction, depending on how good the random sampling was, and the characteristics of a given population.

Nicely put. With a case fatality of ~1%, we need orders of magnitude reduction, not factors of 2. If anyone wants to quibble over a factor of 2 you've missed the point.

Now my nightly rant: I did this kind of false positive calculation in high school statistics, and I hated it. Still do. I hate that we're dealing with this BS. I hate that stupid people put us in this position, and that smart people aren't smart enough to not be absolute knobs.

Why are you citing the NYC study as reinforcement of the other two? It is unlikely that all 3 are correct, because they imply significantly different outcomes of an infection.

Reply on news.ycombinator.com