Hacker News Reader

>"Statistical significance has NOTHING to do with magnitude of difference!"

Here is the equation for a t-statistic (used for the common t-test):

d=mean(a)-mean(b)

s=sqrt(var(a) + var(b))/sqrt(n)

t=d/s

I see the magnitude of difference (d) right in the numerator. The t-value is then compared to the t-distribution to get the tail probability.

View on HN

You are also normalising the difference by the variance, so the t-statistic has no units.

The claim the significance has nothing to do with the magnitude of a difference is just wrong though. It clearly does, this information is just merged with other information about the variance and sample size to get the p-value, which is compared to a threshold to get significance.

I think what the article meant to say was that, for the same number of samples, the p-value when you have delta=50 (the difference between groups), stdev=10 is the same as the p-value of delta=0.5, stdev=0.1.

Depending on the study, finding a delta whose p-value is significant does not necessarily mean that the size of the effect (i.e., delta) might be significant enough to be useful.

it's not "just wrong". On the contrary, it is asymptotically true. In a world of abundant data, that matters.

Please expand on this.

Reply on news.ycombinator.com