Hacker News Reader

Comment by danaris | Hacker News Reader

I'm sorry; I don't follow. Could you clarify?

If the author's setup can only statistically determine effects of 30% or higher, saying "there's no significant change" means "there's no significant change of 30% or more".

Nothing can be said of changes of less than 30%, and that's a hell of a lot of uncertainty. One of the variants could give you 25% more results and yet, those experiments would not see such a result.

As such, it is clear that the experiment is simply not thorough enough to imply the two alternatives are equivalent. Not in the least.

Where do you get that the setup can only determine effects of 30% or higher? I didn't see anything about that in the article, though I admit I only skimmed it.

Simply saying that the statistically significant effect they found was 30% has absolutely no bearing on how large an effect they can detect...

I can't know what is the resolution but neither does anyone else, which is part of the problem. They only show 2 results at 30% (p 0.01) and 25% (p 0.05) which makes me suspect they have a very low resolution. I can't prove it, but I don't need to. They claim results without providing convincing evidence, not me.

Those numbers, by themselves, tell you nothing whatsoever about the resolution of the measures they're using. They tell you solely that for group A, they got a result of 30% with 99% confidence and for group B, they got a result of 25% with 95% confidence. That doesn't mean that they're approaching the limits of their measurable resolution—it most likely means that group B is smaller, or has more variability in it.

If group B had a result of 10% with the same confidence, that would also not indicate that their measures have a resolution of 10%—those things literally have nothing to do with each other. You are reading far too much into the very limited data presented—not merely more than you can prove, more than there can be any possible logical support for.

Reply on news.ycombinator.com