Hacker News Reader

You are also normalising the difference by the variance, so the t-statistic has no units.

The claim the significance has nothing to do with the magnitude of a difference is just wrong though. It clearly does, this information is just merged with other information about the variance and sample size to get the p-value, which is compared to a threshold to get significance.

it's not "just wrong". On the contrary, it is asymptotically true. In a world of abundant data, that matters.

Please expand on this.

I think what the article meant to say was that, for the same number of samples, the p-value when you have delta=50 (the difference between groups), stdev=10 is the same as the p-value of delta=0.5, stdev=0.1.

Depending on the study, finding a delta whose p-value is significant does not necessarily mean that the size of the effect (i.e., delta) might be significant enough to be useful.

Reply on news.ycombinator.com