How big data has created a big crisis in science
theconversation.com
theconversation.com
Big data may bring another dimension to the problem when deep learning will be used in science. However, that's a detail and we're not there yet.
The fact that a paper is reproducible or not is not problematic per se. This is not what defines science. The real problem arises when 1/ a big claim/discovery is made in a paper that is not reproducible, 2/ nobody tries to check the results independently, and 3/ the community takes nevertheless the paper seriously and accepts its findings. All this has nothing to do with the use of statistics (unless the whole community makes the exact same errors) or big data.
Most of the graduate programs don’t require students to take statistics, or if they do, it’s very cursory. Furthermore, students often learn very little about assay design - they end up thinking that non-linear responses are linear and do things like divide assay signals to get ratios (two sins here: assuming the assay response intercepts at 0 and that it’s linear).
So at least for the biosciences, it’s been a shitshow for a while.
I'm asking because there will probably always be something you are not careful about.
So in other words: how bad is bad statistics?
New discoveries that are actually false seem likely to cause a lot of confusion leaving you worse than wrn you started.
https://blogs.sciencemag.org/pipeline/archives/2011/09/22/th...
https://blogs.sciencemag.org/pipeline/archives/2011/12/13/th...
https://en.m.wikipedia.org/wiki/Sirtris_Pharmaceuticals
Famous professor from Harvard discovers possible route to elongating lifespan and then spends 1bn+ in funding to pursue that idea. I haven’t kept up with that story since then, so it’s possible that someone might be able to use it as a therapeutic target.
If the effect size is large, then you can be pretty sloppy with your statistics - in some ways it doesn’t matter because it’s almost a qualitative/binary difference.
Obviously statistics becomes much more important when the effect size shrinks and you are squinting at some data trying to see if a 20% difference is real or not.
Due to the poor foundations in biosciences (everything from the assays themselves, lack of assay replication, errors in interpretation) engineering/optimizing something can be fiendishly hard.
Science would first grind to a halt, reverse itself, then descend into a thousand-year dark age of sophistry, apologetics, and superstition. Alexander Fleming and Jonas Salk would be laughingstocks, Vaccinations would be banned and plagues would visit upon the Earth. Norman Borlaug would be denounced, crop varieties would be chosen by astrology, and famine would be visit upon the Earth. The guilty party for any crime would be determined by Phrenology or Theranos blood tests, the punishment decided by Jungian psychoanalysis, and injustice would rule the Earth.
I only say this because this is more or less what happened during the Mao's Chinese Cultural Revolution, Pol Pot's Khmer Rouge, or the European Dark ages. Historically speaking, it's a very bad sign when science is outright rejected and it most likely means millions of people are about to be murdered, and then millions more are going to starve to death or die of preventable diseases.
And you would be shocked at the lack of statistical education amongst biologists. Often many them freely admit that they chose biology because they like science but are terrible at math.