Comment by danaris | Hacker News Reader

Comment by danaris | Hacker News Reader

Parent Full thread

> The next four are mistakenly reported as "no statistically significant difference in performance." This is a false statement. Having high p-values means that any effect is below the resolution achievable with those numbers.

No...that's what "no statistically significant difference" means.

In these kinds of statistical tests, literally all you can look for is a statistically significant difference. Either you find one, or you don't. Once you know that you've found one, you can say what size it is—a difference of 0.1% is still plenty statistically significant if your p-value is <0.001. It's just not very practically significant.

There are a lot of different reasons you might not find a statistically significant difference, and simply knowing the p-values cannot tell you which ones are true here. That's why a lot of journals (at least in Psychology, the field I have some knowledge about) have been pushing for reporting additional tests and measures.

You're not wrong, but the important point is that people using p-value tests don't really understand what they mean (what practical conclusions should be drawn from results), even when they do do the math correctly; and people consuming them really don't understand what they mean, or even how to know if the math was done correctly.

They mean a lot less than people assume, and in some cases, as you suggest, something different than they assume. (Ie, "statistical significance" relates to confidence that the results showed a reproducible correlation (and are never 100% 'proof' of this, just a level of confidence; and are totally ruined if you've been doing data dreding/p-hacking), rather than any kind of magnitude of effect).

Sure, but "forgetting" that the significance level is a 30% effect is at the least misleading. A real 25% increase or decrease would not have been statistically significant.

I'm sorry; I don't follow. Could you clarify?

If the author's setup can only statistically determine effects of 30% or higher, saying "there's no significant change" means "there's no significant change of 30% or more".

Nothing can be said of changes of less than 30%, and that's a hell of a lot of uncertainty. One of the variants could give you 25% more results and yet, those experiments would not see such a result.

As such, it is clear that the experiment is simply not thorough enough to imply the two alternatives are equivalent. Not in the least.

Where do you get that the setup can only determine effects of 30% or higher? I didn't see anything about that in the article, though I admit I only skimmed it.

Simply saying that the statistically significant effect they found was 30% has absolutely no bearing on how large an effect they can detect...

I can't know what is the resolution but neither does anyone else, which is part of the problem. They only show 2 results at 30% (p 0.01) and 25% (p 0.05) which makes me suspect they have a very low resolution. I can't prove it, but I don't need to. They claim results without providing convincing evidence, not me.

Those numbers, by themselves, tell you nothing whatsoever about the resolution of the measures they're using. They tell you solely that for group A, they got a result of 30% with 99% confidence and for group B, they got a result of 25% with 95% confidence. That doesn't mean that they're approaching the limits of their measurable resolution—it most likely means that group B is smaller, or has more variability in it.

If group B had a result of 10% with the same confidence, that would also not indicate that their measures have a resolution of 10%—those things literally have nothing to do with each other. You are reading far too much into the very limited data presented—not merely more than you can prove, more than there can be any possible logical support for.

Reply on news.ycombinator.com