Comment by martian | Hacker News Reader

Comment by martian | Hacker News Reader

I agree with other critiques here that this A/B testing calculator does little to add to the conversation, and someone who uses it would be mislead in how to interpret results.

The procedure my team uses is:

1) hypothesize an expected conversion rate and whether to use a one-sided test (if we're testing that the state of the world remains unchanged) or a two-sided test (if we're testing that that the state of the world has changed as a result of the variation)

2) run those numbers through this power/sample size calculator to determine the number of visitors we need before we can analyze the experiment: http://www.stat.ubc.ca/~rollin/stats/ssize/b2.html

3) wait for traffic

4) after enough visitors have come to the funnel, pass the resulting conversion numbers through ABBA http://www.thumbtack.com/labs/abba/ [1] to see confidence intervals on our results

For further reading, I highly recommend:

http://visualwebsiteoptimizer.com/split-testing-blog/how-to-...

[1] disclaimer: my colleague wrote ABBA

Hey martian, thanks for the feedback. Your methodology is pretty sound - it looks like we use a similar technique. If you supply a small sample size here, you'll get the message, "There is not enough data to determine a conclusive result".

I also agree. This is effectively 2 lines of R code:

    binom.test(500,2100)
    binom.test(1000,2000)

With a boxplots/confidence interval plots (which are not many more lines). Whether this is useful or not depends on how much people know about statistics I guess.

This calculator makes reasonable assumptions. For example, it doesn't use bootstrapping, and it assumes a normal distribution. I don't see how making these assumptions makes the calculator bad, it just means if you are a stats person who wants to do something different, you'll need to implement your own script. Outside of the HN crowd, there are people who don't code, who nonetheless work with a/b tests, and still need a measure of statistical confidence.

Since a binomial distribution approaches a normal distribution for large N, that's a valid assumption to make.

The arguments in this thread over which is the precise distribution are over analyzing, in my opinion. If I had an experiment that gave one winner with a binomial distribution, and a different winner with the normal approximation, I'd feel little confidence in the results and want to run it again or collect more data. I don't see how that would happen though.

Bootstrapping would be better, sure, but it would be confusing for an a/b test calculator to give you different numbers each time you refresh the page.

You're free to write your own R scripts if you want to do it yourself.

I agree with martian's advice about figuring out a minimum number of visitors you expect to need before analyzing the data, or else you're at risk of biasing the experiment by declaring a winner prematurely.

Reply on news.ycombinator.com