Hacker News Reader

Comment by lliiffee | Hacker News Reader

This error builds on a simpler, even more common one (at least among students): Suppose you have just one treatment (A), and you find that A has no statistically significant effect. This does not show that A has no effect, or that A's effect is small. (It could just mean your dataset isn't large enough.) The error discussed in the article seems to build off this mistake.

View on HN

Could you elaborate on this? Does this assume "you" extrapolate the effect of treatment A to the general population?

I mean, I understand that if you have a sample size of two, find that treatment A does not induce an effect, and conclude that treatment A has no effect [for all test subjects] this does not hold. However, surely if the sample size is big enough (which obviously isn't always clear, but for the sake of the argument let's assume it is) then drawing such conclusions does hold (within the certainty thresholds predefined for your statistical test of choice, such as 99% or 95% probability). Or have I misunderstood?

The issue is that it is nearly impossible to detect "the same". Imagine the sample size required to differentiate between .5 and .500000001. Now, technically speaking, sure-- with any effect (however small) in the infinite data limit a difference would be detected. But this isn't really what these tests are intended for.

The problem is that standard statistical tests have two outcomes: (1) "reject the null hypothesis" or (2) "failure to reject the null hypothesis". Moreover in the most common statistical tests the null hypothesis amounts to something like "the two samples were drawn from the same distribution" so if you fail to find a significant difference you haven't shown they're the same, you've just failed to show they're different. If what you want to do is show that they're nearly the same, you can design a statistical test where the null hypothesis is instead something of the sort "these two samples were drawn from distributions that differ by > X amount" so rejecting the null hypothesis shows they differ by <= X.

Reply on news.ycombinator.com