The reproducibility crisis and other problems in science: John Ioannidis [video]
youtube.com
youtube.com
I have no fucking clue how a soft science study could replicate itself.
I think it is a growing, and problematic, misconception that science is about just reporting facts. It is about adding knowledge...soemtimes that is new facts, but more often it is theories, observations, and questions increasingly at the margins of randomness and probability as our species grows and learns.
How many of those non-reproducible studies showed a robust effect, in contrast to the barely there effect?
speaking openly
that kinda science seems like a bullshit factory (a very lucrative one, of course)
and analogous to an llm hallucination — shallow satisfaction of constraints by enumerating a problem space irrespective of contextual appropriateness, a liability, and forgotten as soon as real value comes along
surely the best science isn't just reproducible by scientists — engineers can get the grubby mits on it, to improve (ok, shape) the circumstances of our lives, and survival
that other stuff is busywork, not "literature". they are very different things...
Probably. We've filled science with Imposters. It seems predominantly women, they are the largest change in science.
The worst non-replicable sciences are also female majority.
Busywork on mice conditions is exactly the false science we are talking about. More excuse why it's not reproducible when mice are mice and are great sub-ins for humans.
"In mice" is the meme create to excuse the garbage that's produced. The studies never originally worked on mice, but now they have an excuse why they don't work on humans.
I wouldn't necessarily blame it 100% on women though. But I see your point.
https://sciencebasedmedicine.org/mistakes/
If you choose to become a policy entrepreneur and your success ends in a Holocaust-scale outcome, you get to shut up, retire, and thank your lucky stars we live in a society that strongly discourages blood debts. That's the deal. No book tour, do not pass go, do not collect $200, and stay the fuck off social media.
And you believe he was wrong and you believe Covid was a "Holocaust-scale outcome" ? He said the rates would be exaggerated and now with the benefit of hindsight, although some would claim common sense after seeing the sensationalism of the media, we can see he was correct. Or do you believe that's not true ?
It might not appear so at the first glance due to Gell-Mann Amnesia effect.
The problem is there is a broken system in place that rewards publishing in high volumes, only getting positive results, and also doing research that supports those in charge. That doesn't mean new ideas don't win out when clear evidence exists but it is the exception.
All of this leads to economic incentives to lie with the data and then you get to scientific papers that people are making decisions based off (like where to put research dollars for Alzheimers) that are fraudulent. This is not good at all.
I actually find the reproducibility crisis you're mentioning to be both more problematic and to receive far less attention than the other. More problematic because it infects every form of science, even the harder/more fundamental sciences, and because it's way less clear how to fix it.
The reproducibility crisis near the top of the stack is just: "do more science and don't believe results until they've been replicated, refined, and matured."
I've always felt that the psychology/sociology studies of the week I hear about on the radio were likely based on some weak statistics in the best of cases and most likely difficult to reproduce.
Science is timeless and powerful. Scientists are human beings that nominally, preferably, fallibly practice science.
Regarding the comparison between science and folk explanations on the internet, I think it's reasonable to hold science to a higher standard than we hold the general population. If most published research findings are false—as this video claims—and most of what idiots on the internet say is also false, that's neither an equivalence nor a victory for science. On the contrary, it erodes the value of science, both literally and figuratively. Right now we need sources of authority.
The connection with Gell-Mann amnesia is that people overestimate truthiness of what they read online. Combined with reading about [real] issues in science, It might create an impression that scientific findings are relatively at the same level as everything else. My point is even all the troubles science is still a head above despite the perception.
We are blessed as a species that John stuck to his principles - and his thirst for empiricism - during the COVID-19 panic, and supported / encouraged his colleagues to do likewise.
This video is only the first 12 minutes of the talk. The rest is here (though it is possibly semi-paywalled? It let me watch it, even though it said it was going to make me sign up for a trial):
https://iai.tv/video/why-most-published-research-findings-ar...
https://www.dailymail.co.uk/news/article-8843927/amp/Just-0-...
"The voice never lies." --Blind woman speaking to a friend
And his story about his hearing of Theranos is lowkey hilarious. And topical, because he's a Dunning-Kruger true-expert.
Iaonnidis's estimates of IFR in the 0.1% to 0.2% range were much closer to the mark.
His early influential paper before that was way off and said IFR might be even lower, around the common cold.
Around 0.5%-1.5% IFR without overwhelmed hospitals was the common scientific consensus very early on and was more right. Some treatment methods like proning and demonstrated effectiveness of steroids in a certain schedule helped drop things a good bit a few months in, around 30% if I remember.
The 0.1%-0.2% was just bad science, taking medians over countries with lagging statistics reports.
Where did you find 3.4%? Isn’t that an upper bound?
Here's a study from March 2020: https://pmc.ncbi.nlm.nih.gov/articles/PMC7118348/
> Adjusting for delay from confirmation to death, we estimated case and infection fatality ratios (CFR, IFR) for coronavirus disease (COVID-19) on the Diamond Princess ship as 2.6% (95% confidence interval (CI): 0.89–6.7) and 1.3% (95% CI: 0.38–3.6), respectively. Comparing deaths on board with expected deaths based on naive CFR estimates from China, we estimated CFR and IFR in China to be 1.2% (95% CI: 0.3–2.7) and 0.6% (95% CI: 0.2–1.3), respectively.
Please provide specific, contemporaneous examples of the 3.4% estimate and evidence of it being "taken as gospel" and "driving the actual policy."
> A Feb. 28 editorial in the New England Journal of Medicine, co-authored by Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases, took the position that the mortality rate may well fall dramatically. It said if one assumed that there were several times as many people who had the disease with minimal or no symptoms as the number of reported cases, the mortality rate may be considerably less than 1%. That would suggest “the overall clinical consequences of Covid-19 may ultimately be more akin to those of a severe seasonal influenza,” the editorial said.
There are many similar lines.
The “I can be trusted with scientific comprehension” crowd fails comprehension once again!
https://www.usatoday.com/story/news/politics/2020/03/05/coro...
That’s not IFR, and carries appropriate caveats literally in the same press release.
The problem is, that was not his estimate for the IFR. It was his estimate for the CFR.
He was predicting 10k dead in the US, which was off by two orders of magnitude. I don't know that anyone was further from the mark than him.
People need to get over their fertilization of contrarianism. Very often, the consensus view is correct.
Where. Source it.