Comment by gostevehoward | Hacker News Reader

Comment by gostevehoward | Hacker News Reader

These are great! Thank you Evan. Your sample size calculator is wonderful and beats the hell out of the 90s tool I've been using which cautions me that my browser must support "JavaScript" to use it :)

As an alternative to the Chi-squared calculator, people might want to check out ABBA, a tool I wrote here at Thumbtack:

http://www.thumbtack.com/labs/abba/

It shares the visual component and the linkability, two great features you've nailed. It lacks the live updating and the slider, which is really cool and something I've wanted to add to ABBA for a long time. On the other hand, it supports multiple groups compared against the baseline simultaneously and incorporates a correction for multiple testing into its p-values and confidence intervals, which can be handy. It also uses different mathematics under the hood, but that's not going to be a concern for most users.

Glad to see another step towards a more statistically-aware world!

You'll be interested to hear that the digital analytics team at the Obama campaign sometimes made use of your tool for sharable A/B calculations.

I even built a bookmarklet for quickly grabbing numbers off off of a page (usually for Google Analytics data) and passing them to your calculator. https://github.com/yahelc/ABBA-bookmarklet

Wow this bookmarklet is awesome! I'm going to start using it. Thanks!

Small follow-up question: I'm curious how your sample size calculator chooses its value. Traditional power analysis asks for two proportions rather than for a proportion and an effect size. So the sample size for a positive effect differs from that for a negative effect (from the same baseline). I'd imagine your tool would conservatively present the larger of the two. However, it seems to present something near the midpoint of the two. Is that the intention? Or is there some other statistics being used here?

(For example, with an 8% baseline conversion rate, a 1% absolute detectable different, 85% power and 10% significance level, your tool says 10,583 per branch. R's `power.prop.test` gives sample sizes of 11,182 for a positive change (8% vs 9%) and 9,974 for a negative change (8% vs 7%). The exact midpoint is 10,578.)

Reply on news.ycombinator.com