A/B Testing Rigorously (without losing your job)
elem.com
elem.com
1. Rigorous (this one) - How to use frequentist techniques to define a very rigorous statistical procedure that - no matter how small the bias in the test - can give correct answers with very high probability (and, of course, support multiple looks at your statistics).
2. Fixed data - Your business has an upper limit on how much data it realistically can collect. How to design a statistical procedure that makes reasonably good decisions while remaining within that limit.
3. Simple test - How to use Bayes' Theorem to design very simple and straightforward test criteria, where every A/B test has a known cost up front.
4. Bayesian statistics - How a Bayesian approach compares to the previous ones. Where the sticking points are, what its advantages are.
5. Bayesian optimization - How a Bayesian approach can let you turn the problem of A/B testing into a pure optimization problem where your decision to stop is based on whether the cost of continuing the test exceeds your expected return from continuing it over some time horizon.
We'll see how far I get. As it goes on, they become tricker to write at the level that I'm aiming for. And it may be that people will be interested in something else. But I thought that the first two were important. If for no other reason than to give me something definitive to point people at who read Evan Miller's article and then were telling people to run A/B tests in a way that made no business sense.
> We will follow Evan's lead and use a frequentist approach.
I though: "how about using the correct approach instead?", followed by a helpful comment about how you are a Bad Person¹. Hem.
Then I noticed that the way you talk about p-value doesn't sound very frequenty. And then your comment. I eagerly await for the rest of the sequence –err, series².
[2]: Come to think of it, this does looks like LessWrong material to me.
I know how wonderful it can be to feel that you have realized The Truth, but the frequentist school of statistics does not remain in existence just because statisticians are too stupid to recognize the obvious superiority of Bayesian methods.
I mean, many frequentist methods directly contradict Cox theorems! That should convince anyone they belong to History, not Science, doesn't it?
So. Who in her right mind would reject the three assumptions behind Cox's theorem, and for what reason? Assuming we don't reject those axioms, why should we not reject Frequentism at once?
But if I finish, I'll probably have learned a lot about when to use each. Hopefully because I figured it out correctly, and not because I said something egregiously wrong and got corrected by someone who knew better!
I think a lot of people improperly understand the Frequentist/Bayesian divide; both look at likelihood functions but the Frequentists assert, as a matter of experimental interpretation, that the likelihood function should depend only on the observed data while the Bayesians are willing to admit "out-of-experiment" modifications to the likelihoods implied by the data (i.e. the "prior").
Frequentist arguments correctly point out that the prior is, from the isolated standpoint of the experimental observations, purely subjective and non-falsifiable. Bayesian arguments correctly point out that experiments are designed by humans in context, and that the context may imply a prior distribution that is not evident within the observations themselves. Neither is incorrect, but they answer different questions. Which question is the right one to answer is context-dependent.
As a concrete example, consider an A/B test where A and B perform very similarly. Merely seeing A's performance against B does not carry the contextual information that A has consecutively beaten one hundred previous non-B competitors that sort-of-look-like B. The Frequentist will correctly argue that, if you don't explicitly model the A/B test generation process (as opposed to just the A/B test itself), then all the data can tell you based purely on the laws of probability is that A and B are likely to be pretty similar. The trouble faced by the frequentist is that it's pretty easy to model an individual A/B test for, say, binary outcomes, but it's very hard to explicitly model the generation of A/B test competitors. So the Frequentist says "all I can tell you and still be strictly objective is that A and B look similar".
The Bayesian, rather than explicitly modeling the generation of tests as the Frequentist would like, wraps up that contextual knowledge into a subjective prior distribution that heavily weights A over B simply because A has won so many times before. The Bayesian's advantage is that the previously-blocking part of the probability modeling problem is punted into a subjective, experimenter-designed, input. This is a tradeoff, not an increase of correctness! It will "do the right thing" assuming the new B looks a lot like the old Bs, but will do the wrong thing if the new B is dramatically different and maybe looks a lot more like A with small tweaks.
Bayesianism is not a universal solution to probability modeling problems, and Frequentism is not an obsolete unhip method.
Personally I'm eagerly awaiting #5: treating the question of whether to continue a test as an optimization problem.
I'll be waiting eagerly to read the rest.
Cheers!
To provide an example, you run a test and find Page 1 outperformed Page 2 by a factor of 1.1, but that was non-significant at your desired significance level (say, p=0.05 and power = 0.8). You should deploy page 1 instead of page 2, assuming there are no other costs associated with deploying page 1 instead of page 2, because your BEST GUESS RIGHT NOW is that page 1 is 1.1x better than page 2.
There are two schools of thought about the approaches to sequential testing, the Bayesian approach lead by Anscombe, and the frequentist by Armitage. I talked a bit about this and outlined Anscombe's approach here [1]. And it is great to see such a nice write up of the frequentist approach and the tables of the stopping criteria
[1] http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-te...
If all goes perfectly, I will discuss more ways to think about the problem than just those two, and try to show some connections that may surprise people. But, judging by your nice article, I doubt that I'll prove to have anything to say that you don't already know. :-)
The issue you will run into here is that 95% confidence means that you will only have a false positive 5% of the time. It does not mean a neutral finding is 95% likely to be neutral. The lever that controls that is statistical power, which is oft-ignored in conversations about A/B testing. Most statisticians use 80% power, which means a full 20% of neutral findings were false negatives.
From experience I've learned this puts you back in the gut-based guessing game those employing these methods are trying to escape.
[1]: http://www.exp-platform.com/Documents/2009-ExPpitfalls.pdf
Why not do a one-sided test where the null hypothesis is "A is at least as good as B" and the test hypothesis is "B is better than A"? It seems like you'd be gaining some more power to detect B being better, without losing much since A's safe to choose regardless of if they're about the same or if A's actually better?
The point of the null hypothesis is that it is an absolute worst case. So whatever the real difference, it can't be harder to detect than the null hypothesis. Therefore if we make very few mistakes on the null hypothesis, then we're confident that we'll make very few mistakes, no matter what.
And finally you're right that going with the green button is better than just tossing up your hands. But how to decide how much better is a surprisingly complicated question, and there is no simple answer about how to quantify it. (Remember, statistics has to be done with numbers, so quantifying the answer matters.)
It should let you get away with a lower sample size at the expense of possibly only being able to conclude "A's better" or "A's not better" instead of "A's better" or "A and B could be the same" or "B's better." (It's unclear to me what would be the problem with looking at the P values for both the > test and the != test, for a single test, and only "falling back" to the > test if you happen to be in a range where you can say "we can't say for sure that B's better, but we can be pretty sure that A isn't better," like how the different P values are presented here[1].)
It still wouldn't be valid to do something like "oh the two-sided one was inconclusive but leaned in favor of A, so let's do a one-sided one to check if A's better after all," though.
[1] http://www.ats.ucla.edu/stat/mult_pkg/faq/general/tail_tests...
In the next article I plan to relax things a lot farther than just one-sided versus two-sided tests!
I haven't come up with a decent answer yet especially for smaller sites that can't run tests on very specific segments...
When you make decisions at 95% confidence, your guarantee is that at most 5% of the time are you going to wrongly conclude that one is better when it isn't. However you have absolutely no guarantees about having correctly called the direction of the test if you do hit that significance level. (Indeed if the null hypothesis is correct, every time you call the test, you're wrong!)
What you did is simulated the test many times, ignored the cases where you were told there was no answer (thereby throwing away a large part of your guarantee), and found that you could be wrong a large portion of the time. Furthermore if there was a large, discoverable, random factor, you found that could be correlated with a lot of the mistakes. Unfortunately, in addition to the discoverable factor, there are lots of unknown random factors that also get randomly correlated. And even if there aren't, there is always the possibility of experiencing bad luck.
The only way to solve this is to throw enough traffic at the problem that the underlying bias can be reliably detected statistically. There is a complicated relationship between the size of bias you're willing to get wrong, and the amount of data that you need to collect to reliably detect it. I hope to explore that relationship in the next two articles.
For larger sites, that solution is perfectly workable. For smaller ones it is not, and the best that I can suggest is that they rely heavily on design principles that have been validated through A/B testing on larger sites, and hope they are not going too far wrong.
I was mobile when I wrote the original question - perhaps a better way of phrasing it would have been something like:
Don't many (most? all?) of these theoretical approaches assume that the sequence of results for each page (call them a_i and b_i for i=1,2,3... where each x_n is 0 [no conversion] or 1 [conversion]) are sequences of iid random variables with underlying conversion probabilities p_a and p_b? In reality, these sequences are much more complex and, if the scale of the variation in conversion probability within the sequence is greater than the difference between p_a and p_b won't the test be much weaker than we originally thought?
To use an example that is simplified vs. reality but hopefully indicates what I mean, imagine that we have two traffic sources - one with a conversion probability twice that of the other (on each page variant - so p_1_a = 2p_2_a and p_1_b = 2p_2_b). We randomly send traffic from both sources (1 and 2) to each variant (a and b). Do the standard tests work even though our sequence of conversions are not iid?
One good way to see why is this. The theoretical approaches that you're talking about do not care about the internal details of your random number generator. You could have p_a be the result of a single random decision (convert or not), or be the result of first finding out that we had a random source, then having conversion probabilities that depend on the source. Either way, as long as in the end you get a stable p_a of converting from noticing you have a new visitor to an actual conversion, probability theory says that the exact same statistical statements will be true, and you'll have entirely equivalent results.
(Note, the way I set up my particular approach means that actual conversion rates can shift over the test without invalidating the results. I didn't call that out too strongly, but I value that detail.)
However there is a gotcha lurking. The gotcha is that if conversion depends on your source, then A/B testing will optimize for your current traffic mix, but its result may become wrong if that mix shifts. My usual approach is to simply assume that the world is not going to be malicious in this way, unless I have specific reason to suspect it is.
Thus, for instance, I would not think twice about source vs button color. But if I had a landing page with random testimonials, I'd expect that a testimonial from pg would convert better than a testimonial from Phil Ivey. And conversely for traffic from a gambling site.
> Therefore we're faced with a series of decisions, at each number of conversions n, m more turned up A than B (m can, of course, be negative).
which I am not able to parse.
I think would be another way of saying it. I could be completely wrong though, it is a bit confusing.
If so, how would you like to be acknowledged?
Waiting for rest from the series!
Grzegorz
If you stop your tests on the p-value cutoffs that I gave in my third graph, you'll get very strong a priori guarantees that the decisions that you make will be right. The downside is that you won't have any guarantee of making decisions in reasonable time. But that is the subject of the next article.