Simple Sequential A/B Testing | Hacker News Reader

Simple Sequential A/B Testing

evanmiller.org

59 pointsrevorad9 comments

A much simpler approach is to AABB test instead of AB test. Rather than splitting your users into 2 buckets (A and B), split them into 4 buckets (A1, A2, B1, B2). Give groups A1 and A2 one variation and groups B1 and B2 the other variation. When A1 equals A2 and B1 equals B2 then you know you have statistical significance and you can compare A1+A2 to B1+B2.

This is great advice. One of the best things about doing AABB testing is when your two A groups & B groups don't converge, you can identify bugs in your testing procedure or measure the margin of error (since you know those groups are seeing the same thing and should be performing identically). Seeing two identical A groups with wildly different results will make you more skeptical of generic A/B results & make you more rigorous about your testing.

> since you know those groups are seeing the same thing and should be performing identically

That's not how A/B testing works. 95% confidence means you should expect a 5% false positive rate, i.e., you should expect the difference measured in an A/A test to be statistically significant 5% of the time. You'll always measure some difference, since no two random samples will be 100% identical in every regard.

The procedure you and the parent propose is tantamount to selecting 1 out of every 20 test results and discounting it for no real reason. It adds extra cost to your A/B testing without producing more reliable results.

See also: https://xkcd.com/882/

It's a different matter if you're running multiple A/A-type tests over an extended period of time to ensure that the false positive rate is actually 5%, a kind of meta-statistical test. As a sanity check this is sound, but vastly more expensive than what the OP is proposing (for example). I've never seen anyone use A/A, A/A/B, A/A/B/B, etc. tests in this way. Rather, I've only ever seen them used as you and the OP suggest: the two A buckets should be "the same" and if they aren't, the results should be thrown out.

That's a good strategy, but is solving a different problem. You still have to collect N samples. The proposed strategy is useful for situations where each additional sample has a high cost and being able to terminate early, if possible, is highly desirable.

Doesn't sound very rigorous. How do you know when A1 and A2 are "equal"?

To be honest this method sounds a lot simpler. You only have to track two numbers!

I prefer AABB testing because it doesn't take me much time to figure out if the test needs to run longer and you get used to the patterns your conversion rates follow pretty quickly in tools like VWO. I've talked to some large CRO shops who do the same thing, as they are working with several accounts at the same time and need to be able to tell quickly how things are going.

Then again, CRO is only a piece of my job and I have lots of other stuff to look at everyday, so anything I don't have to expend too much mental energy on, but can still trust, is a nice convenience. If I was optimizing full-time, I'd be a lot more smart/clever in how I went about it.

You would need to propose a method to compare the confidence intervals of A1, A2 and B1, B2 without hurting our target coverage probability on the outer confidence interval. This is starting to sound really complicated.

Why not just estimate p(A - B | observed data) and be done with it?

Because that's a lot harder. At VWO we do exactly this in our new SmartStats tool - we believe Bayesian stats are far more intuitive to the end user. But we've got a whole compute cluster running these calculations 24/7. For comparison, our old frequentist calculations were done in PHP at page render time.

A formula that can be easily plugged into Excel does have value over a formula that requires someone to write a python script and do a monte carlo simulation.

Evan's method is also nice in that it doesn't require a lot in terms of knowing your (possibly very small) conversion rate ahead of time.

Reply on news.ycombinator.com