gostevehoward

Wow this bookmarklet is awesome! I'm going to start using it. Thanks!

Small follow-up question: I'm curious how your sample size calculator chooses its value. Traditional power analysis asks for two proportions rather than for a proportion and an effect size. So the sample size for a positive effect differs from that for a negative effect (from the same baseline). I'd imagine your tool would conservatively present the larger of the two. However, it seems to present something near the midpoint of the two. Is that the intention? Or is there some other statistics being used here?

(For example, with an 8% baseline conversion rate, a 1% absolute detectable different, 85% power and 10% significance level, your tool says 10,583 per branch. R's `power.prop.test` gives sample sizes of 11,182 for a positive change (8% vs 9%) and 9,974 for a negative change (8% vs 7%). The exact midpoint is 10,578.)

These are great! Thank you Evan. Your sample size calculator is wonderful and beats the hell out of the 90s tool I've been using which cautions me that my browser must support "JavaScript" to use it :)

As an alternative to the Chi-squared calculator, people might want to check out ABBA, a tool I wrote here at Thumbtack:

http://www.thumbtack.com/labs/abba/

It shares the visual component and the linkability, two great features you've nailed. It lacks the live updating and the slider, which is really cool and something I've wanted to add to ABBA for a long time. On the other hand, it supports multiple groups compared against the baseline simultaneously and incorporates a correction for multiple testing into its p-values and confidence intervals, which can be handy. It also uses different mathematics under the hood, but that's not going to be a concern for most users.

Glad to see another step towards a more statistically-aware world!

I'm impressed with how nicely this handles the whole A/B testing stack. But the statistics have some potentially important shortcomings -- I posted a github issue about it:

https://github.com/maccman/abba/issues/3

(Full disclosure: I authored the ABBA library mentioned by matsiyatzy, so this is incidentally self-promotional. But I wrote that tool with the hopes of improving the use and interpretation of A/B test statistics in the wider community :)

Well either the app object is global, in which case you've got global mutable state, or you're defining your handler functions later than module load time, which is pretty uncommon practice.

Some might say it's a dangerous abuse of decorators.

Since decorators (generally) run at module load time, any stateful decorator (usually) implies the use of global mutable state, which is (considered by many to be) the root cause of much bad design, convoluted flow, limited reusability and untestability. This is perhaps why most decorators in the standard library are pure (off the top of my head).

This is well beyond the scope of the post but an important and often overlooked point in my opinion.

Theirs has a nice, simple presentation which makes it very easy to use, especially for casual users. But it includes much less information (no confidence intervals and p-values are only approximate), uses much simpler statistics (z-test with no multiple testing correction can result in highly overconfident results), and doesn't describe what underlying statistics it uses. Although not as immediately accessible, I think Abba has much more to offer to anyone who's serious about A/B testing.