Hacker News Reader

Comment by WA | Hacker News Reader

and tells you how long you’ll need to run your experiment for to see statistical significance.

I always thought that statistical significance isn't something that should be tried to achieve, but merely a performance indicator how good the experiment was. Isn't it odd to try to "achieve significance over time"?

Shouldn't it be: "Your experiment requires 5,000 visitors and after that we'll check if the result was significant enough to not be merely due to random chance"?

Could someone with more statistical understanding elaborate this a bit?

View on HN

> Shouldn't it be: "Your experiment requires 5,000 visitors and after that we'll check if the result was significant enough to not be merely due to random chance"?

That's basically what is happening with the tool, I think. It is asking for how many users per day you get in order to approximate the sample size for x days, then it's asking how much power you want. Power is the likelihood of detecting a difference if there is one. It also asks what confidence level you want. All of those together give you an approximate answer to the amount of time, assuming # of users/day is roughly constant.

There are 4 inputs needed to estimate sample size for a test: power, confidence level, expected difference, and variance. You need all 4 before you run any test. You use the A/A test to estimate variance. Power is the probability of detecting an x% difference when one really exists. Typically you see .8 or .9. Confidence level is the probability of detecting a difference when one really does not exist, typically .05. The 4th item is the expected difference of the test. If you want to detect a 1% difference, you will need more sample than if you want to detect a 5% difference.

You have to know all 4 before you do a test. A test is designed specifically to detect a certain difference. You cannot launch a test without knowing that as part of your hypothesis.

Yep, that's a more precise version of what I was saying w.r.t. estimating sample size. I think the tool makes some assumption about variance, but the other 3 are things you supply. Note that I wasn't saying anything about the A/A test article, just the sample size estimator that's linked to.

Reply on news.ycombinator.com