Comment by nonbel | Hacker News Reader

Comment by nonbel | Hacker News Reader

Parent Full thread

Null hypothesis significance testing. It is a "method" of statistics that resulted from confusion between Ronald Fisher's significance testing with Jerzy Neyman and Egon Pearson's hypothesis testing sometime in the 1940s. None of those three are free of blame either.

While there is all sorts of things wrong with NHST, the primary thing is that researchers never actually test their own hypothesis. Instead they test a "null" hypothesis that acts as a strawman, usually that some parameter equals exactly zero. Then when the strawman is "rejected", they go on to say the results support whatever theory/idea it is they favor.

Rather than come up with precise predictions deduced from their idea, they leave them vague (If I am right, A will be correlated with B). Then calculations are used to reject the very precise strawman, giving an entirely false sense of rigor to a handwavy, nonsensical ritual.

It is very common, and seems to get more and more popular every year despite all efforts by scientists and statisticians to stop it (because it lets you seem scientific about proving whatever you want). I was trained to think that way myself not too long ago, with absolutely zero mention of any controversy.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4347431/

http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf

http://www.fisme.science.uu.nl/staff/christianb/downloads/me...

http://andrewgelman.com/2016/09/30/why-the-garden-of-forking...

>Instead they test a "null" hypothesis that acts as a strawman, usually that some parameter equals exactly zero

How is "there is no difference between my groups/conditions" a strawman?

This is by far the most common H0.

Because, except in rare cases, the model/theory/idea of interest does not predict "there is no difference between my groups/conditions". It predicts something else. In the rare cases a zero difference is predicted, the method is scientific.

For nearly all use cases many plausible models/theories/ideas will predict "not exactly zero difference", so rejection of the hypothesis "difference = zero" is useless to the scientist. Mathematicians apparently don't get this, which is what inspired the Fisher paper I linked to.

And yes, it is the most common H0. No argument there. The problem is so widespread and destructive that the mind instinctively rejects its existence as unbelievable.

Sorry, I still don't understand. Why would an absence of effect predict anything other than a zero difference (assuming infinite samples)?

No one predicts "an absence of effect". Don't worry, it took me years of reading the literature, running my own simulations, and having discussions in person and online. It is really difficult to accept how bad this is.

Please stop being condescending and actually explain yourself.

>No one predicts "an absence of effect"

In my field, we most certainly do predict an absence of an effect, in the following sense:

1. if a manipulation does not have an effect on the system, then we expect no systematic deviations in the measurement distribution

2. point 1 will be revealed by H0 (no systematic change in measurement distribution across conditions) not being rejected.

It's beginning to sound like you've encountered people who misuse or misinterpret this technique -- and I agree that there are alarmingly many such people -- but this is different from the technique being invalid.

Reply on news.ycombinator.com