How Priming Research Went off the Rails (2017)
replicationindex.com
replicationindex.com
You just have to think for yourself and weigh the evidence, be humble and open to other perspectives. Mindlessly following perceived experts won’t save you.
What a gem to read, absurt funny title "The Grad Student Who Never Said "No".
And the fact that at least 54 times people took a snapshot of the webpage.
We should print this on toilet paper and sell it to universities.
It doesn't sound likely to me, but neither do a bunch of other cognitive results (the gorilla running through the basketball court comes to mind).
I'm all for questioning and rerunning tests, but dismissing them because they seem clearly fanciful without data just brings us back to pre-science. Science which is only allowed to discover plausible-sounding things isn't science at all.
There's an interesting chapter in Jaynes' Probability Theory: the Logic of Science where he talks about how much evidence it would take to convince him that ESP is real. At the time that book was written, there were experiments supporting it that failed to replicate.
Basically you're totally allowed to read a scientific study, say "okay, that's an interesting result", and then still not believe the conclusion if you have enough reason to doubt the premise a-priori.
> disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true.
It's as if they know they're wrong, but are trying to force themselves to believe it any way.
That's why with the project I volunteer for (Plaudit.pub) researchers recommending research have to explicitly indicate whether they think a work is robust and/or exciting, where neither block publication nor the recommendation.
If you look at the unfolding replication disaster in epidemiology, re: Ferguson and his totally non-replicable Report 9, it's really turning over the rocks on an academic culture in which:
• Academics flat-out refuse to question each other across disciplines, practically as a matter of honour.
• Nobody else being able to understand your research is something boasted about on Twitter and in the press rather than being a source of embarrassment.
• Scientists routinely refuse to reveal their data and code, making a mockery of both peer review and replication as a process.
• Anyone who does engage in critical thinking about their work gets shut down, censored, attacked as an "armchair epidemiologist" etc - often it seems by people with academic backgrounds.
After years of non-stop hiring, tech firms are now flooded with naive post-docs steeped in academic culture, who appear to have totally bought into the notion that critical thinking about the output of "experts" is not only morally wrong but outright dangerous and needs to be suppressed. This has led to the absurd sight of Google/Twitter trying to delete anyone who disagrees with the WHO at a time when the WHO routinely disagrees with itself as of a few weeks ago, or where the WHO disgrees with the CDC, or where experienced doctors disagree with both, and it ends up being the doctors that get shut down because they are mere doctors and not "experts"!
It feels like this whole conflation of academia with "expertise" is leading society to disaster; why should any of us believe universities when they say they teach critical thinking? Someone trained in critical thinking wouldn't just accept such a claim at face value, they'd demand a weight of evidence. But no such evidence is ever presented.
The landmark of bad epidemiology is of course Andrew Wakefield, who almost single-handedly destroyed the effectiveness of measles vaccination.
Experts are in trouble. Some of the "non-experts" are right. But as often as not it's the confidently extremely wrong who get promoted by the media and social networks.
Over time it seems like the gap between the cranks and experts keeps closing. How cranky are these cranks, really?
You mention vaccination. Anti-vaxxer has pretty much become a byword for dangerous quack science denier. It conjures up stereotypes of soccer moms who are certain their special snowflake child can't take the "risk". But vaccines are perfectly safe; doesn't everyone know that?
A few weeks ago I got a reply to a comment of mine here on Hacker News that mentioned Pandemrix, a treatment I'd never heard of:
https://news.ycombinator.com/item?id=22922864
DanBC, who I often end up debating with on these forums as we are somewhat different in political outlook, remarked "Pandemrix causes narcolepsy in 1 in 60,000 people who take it. (so, for the entire US that's about 5000 with a life long debilitating illness that requires constant care.)"
At the time I replied that I'd not heard of this and without doing more research couldn't really comment. I took his claim at face value and didn't think much more of it, filing it away to look at later. Well, turns out this claim is both completely true and much more importantly, to my surprise Pandemrix is a vaccine developed for the swine flu:
"In August 2010, The Swedish Medical Products Agency (MPA) and The Finnish National Institute for Health and Welfare (THL) launched investigations regarding the development of narcolepsy as a possible side effect to Pandemrix flu vaccination in children,[5] and found a 6.6-fold increased risk among children and youths, resulting in 3.6 additional cases of narcolepsy per 100,000 vaccinated subjects"
https://en.wikipedia.org/wiki/Pandemrix
So, erm, yeah. About those anti-vaxxers. Who think vaccines might sometimes be dangerous for their children. And who get laughed at and shat on constantly. And shut down. And censored. And told they don't understand the science. About them.
Damn.
These people never really get taken seriously regardless of what claims they're making. The moment a position starts being labelled quack/crank/dangerous misinformation, you're already at the point of "coarse" characterisations, or mis-characterisations.
So now look what's happened. Apparently there are two kinds of anti-vaxxer, the ones worried about "proven risks" and then the real cranks making "paranoid claims". But the sort of anti-vaxxers who would have been refusing to give their child Pandemrix before it was understood to be linked with higher narcolepsy rates would have come across as paranoid, wouldn't they? They'd have had to say something like:
"I don't know. It's a new vaccine. It might be dangerous. I'm not sure swine flu is dangerous, I'd rather not vaccinate my child."
and they'd have got an answer from us HN-reading types of the form:
"Swine flu is very serious, the WHO has declared a global pandemic and clinics are being overwhelmed with infected people. Vaccines are well understood and very safe. Experts say the MMR scare was just debunked crank science. If you don't vaccinate your child you're an ignoramus who is putting both your child and other people at risk, you shouldn't even really have a choice. I will report your tweets to Twitter for misinformation."
And we'd have been wrong and they'd have been right.
https://en.wikipedia.org/wiki/2009_swine_flu_pandemic#Histor...
"The Mexican government closed most of Mexico City's public and private facilities in an attempt to contain the spread of the virus; however, it continued to spread globally, and clinics in some areas were overwhelmed by infected people ... In late April, the World Health Organization (WHO) declared its first ever "public health emergency of international concern," or PHEIC,[38] and in June, the WHO and the U.S. CDC stopped counting cases and declared the outbreak a pandemic"
I think I just gave you an example. Anyone who objected to Pandemrix on the principle that they were an anti-vaxxer would have been classed as a paranoid anti-science crank right up until it turned out, whoops, there actually was a problem with it, at which point they magically changed into people with "rational concerns".
That's obvious now but it wasn't at the time when it actually mattered: at the moment children were being injected with it.
How many anti-vaxxers have you talked to personally? For me I must admit the answer is none. I know these people exist only through media reports. I no longer trust the media, especially when it comes to the matter of "experts" and their reliability, so that's already a bad start. So are you sure MMR is their only reason for concern, or is that a "crude mis-characterisation" designed to dismiss them? There was an anti-vaxx movement in the USA before Wakefield so it's unlikely that's the only thing that motivates them.
Heck, how sure are you about the details of the whole MMR incident? Was Wakefield truly discredited? Remember that as far as the official establishment is concerned Professor Ferguson is still a reputable expert with an excellent reputation, and I definitely don't believe that, so apparently what the academy thinks of scientists isn't all that reliable.
We lowered the amount of regulation in order to release pandemrix because of the pandemic. We see (even on HN) calls for a similar relaxation of regulation during covid-19.
Pandemrix did cause harm. It caused severe harm. But it was withdrawn because of the harm it caused.
The people opposed to vaccination are not saying "vaccination is mostly safe but rarely causes harm", they're saying "vaccination is massively unsafe".
Narcolepsy caused by pandemrix affected about 1 in 50,000 people who had the vaccine. People opposed to vaccination aren't talking about narcolepsy. They talk about autism -- and there are very many studies that show that vaccines do not cause autism.
And all of those studies showing safety are in addition to the uncovering of the deception and unethical practice that Wakefield was involved in.
And don't forget that Wakefield wasn't anti-vaccination. He was developing his own measles vaccination, and he had to show that MMR was dangerous if he had any chance of getting his own vaccine approved and used.
They fail to trade off the risks appropriately.
No one can claim vaccines are "safe". They induce a host immune response so that the immune system is primed to fight a subsequent infection. The immune system can cause damage to the host, which is why we have a whole catalogue of autoimmune disorders, many of which can be modulated by infectious diseases and parasites. There is a risk a vaccine could induce a host immune response which causes damage. But it's a small risk.
You have to trade off the risk of vaccination with the risk of being infected with the disease, and the risk to society at large through spread of the disease when vaccination rates are too low.
In pretty much all cases, the risk of the disease is many orders of magnitude higher than the risk of the vaccine. If there's a 0.000001 chance of the vaccine causing problems vs a 0.0001 of dying or suffering long-term damage from the disease then the choice is obvious: get vaccinated.
Unless the "anti-vaxxers" are basing their arguments upon quantitative data, then I'll continue to view them as crackpots with little knowledge or understanding.
(I'm an immunologist, by the way.)
Of course not, but they do anyway as I'm sure you must know!
Imagine someone who is just starting to worry about vaccines does a google search and ends up on this top result:
https://en.wikipedia.org/wiki/Vaccine_hesitancy
It says: "Arguments against vaccination are contradicted by overwhelming scientific consensus about the safety and efficacy of vaccines.[5][6][7][8]"
The first citation for this claim is an article on the WHO website which quotes Dr Giovanni Rezza, Director of the Department of Infectious Diseases at Italy’s National Institute of Health. He says:
"The media often sets up a false opposition between public health officials and anti-vaccination campaigners, rather than conveying a clear message that there is an overwhelming scientific consensus in favour of nationally recommended vaccines"
Scientists are quite happy to insist that there's no debate to be had at all; this is very far from it being all about relative risks which reasonable people can disagree on.
Unless the "anti-vaxxers" are basing their arguments upon quantitative data, then I'll continue to view them as crackpots with little knowledge or understanding.
I assume any anti-vaxxer with any knowledge would cite a case like Pandemrix, which does have quantitative data. In fact DanBC provided some: maybe 5000 long term cases in the USA if everyone had been vaccinated. But I think you must admit, clear data is a luxury that nobody seems to have about viral spread. The numbers for how many people got Swine Flu vary dramatically depending on who you ask, even today. If I were asked to support an anti-vaxxer in a debate, I'd point out there's no reason to believe scientists can accurately weigh the costs and benefits of vaccines given their inability to even tell us accurately how many people die of the flu each year, and the fact that diseases caused by vaccines only become widely accepted years after the fact.
The problem is how do I trust these numbers. That the vaccine risks aren't underestimated and the disease damage over-estimated. Something that can be confirmed only when enough time/data has elapsed.
The current pandemic is an example. Almost everyone who publishes data has some motive to push a viewpoint (knowingly or unknowingly). It will take some time before one can trust the data and its inferences. But then meanwhile, one is being asked to trust the experts who are not open about the data and methodologies.
That's not really true. In fact, it's practically a trope that physicists and economists blithely wade into other fields; there's even an xkcd about it (https://xkcd.com/793/). However, it often turns out that people in the original field do actually know relevant stuff, which makes it hard to do this well.
For COVID, I wanted people outside the field not to shut up entirely, but to tread lightly: before trumpeting a "cubic model" or "quadratic spread" from the rooftops, do some digging to see why this very simple solution hadn't occurred to anyone else.