Comment by Eliezer | Hacker News Reader

Comment by Eliezer | Hacker News Reader

Dear Optimizely: Your statistics do not tell you that B has a 95% chance of being better than A. Your statistics tell you when the excess of B over A has less than 5% probability, assuming B and A are actually equally effective.

A Bayesian would understand this in terms of prior probabilities and likelihood ratios, but to put it into nontechnical terms, suppose that you tried out 15 different alterations and none of them seem to work. Then on the 16th, your detector goes off and says, "Less than 5% probability of these results arising by chance!" Do you conclude that it's 95% likely that this version is genuinely better? No, because the first 15 failed attempts told you that improving this webpage is actually pretty hard (the prior probability of an effective improvement is low), and now when you see that the 16th attempt has a result with a less than 5% probability of arising from chance, you figure "Eh, it's worth testing further, but probably it is just chance."

Another extremely important point is that the classical statistics you learned to use to decide that something was <5% likely to arise by chance, only apply if you decided in advance to do exactly that many trials and then stop. Your chance of finding, on some trial, that your running total of results is "statistically significant", when A and B are actually identically effective, is considerably greater than 5%. See http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ... - a trial I ran with 500 fair coinflips had at least one step where the cumulative data "rejected the null hypothesis with p < 0.05" 30% of the time.

You're not really to blame for this mistake, because the horrid non-Bayesian classical statistics taught in college are just about impossible to understand clearly; but it does sound to me like someone at your org needs to study (a) Bayes's Theorem (b) the case for reporting likelihood ratios rather than p-values (likelihood ratios are objective, p-values decidedly not) and (c) the beta distribution conjugate prior (which would make progress toward having priors and likelihood ratios over "These two pages have a single unknown conversion rate" or "These two pages have different unknown conversion rates"). Or in simpler terms, "Someone at your company needs to study Bayesian statistics, stat."

[I'm one of the co-founders of Optimizely] Hi Eliezer, thanks very much for your thoughts--these are great. You're absolutely right that classical p-values are not without their shortcomings, which can be significant when misused! A Bayesian approach would address some of these shortcomings, but would also introduce other hurdles, like generalizing the selection of a prior distribution and, more importantly, (IMHO,) distilling likelihood ratios in a way that can be quickly grasped by someone with no formal stats/math background. In simpler terms, classical statistics is not perfect, but in most cases it provides a lot of useful information in an easy-to-grasp way.

That said, we've started looking more seriously at incorporating Bayesian techniques into our interface and would love to get your thoughts on great ways to communicate these concepts to our customers. I've reached out to you directly off-thread and would love chat further!

The p-value represents "Probability this is all luck". So why isn't 1 - p "Probability this isn't all luck"?

>> Another extremely important point is that the classical

>> statistics you learned to use to decide that something

>> was <5% likely to arise by chance, only apply if you

>> decided in advance to do exactly that many trials and

>> then stop.

I agree. Not doing that I'd say it's "fudging with the numbers". I can't find on their article where they did this though.

A better interpretation might be "probability this would happen if results were governed purely by chance". Note that the distinction is important if the process is in fact governed by chance!

Lets say I roll a dice two times and get a six both times. The probability of this happening is 1/36 or about 3%.

Would you say I have established with a 95% confidence interval that the particular die I'm using always rolls six? No, because you have good reason to believe that the the results are in fact random. Or in other words, you have a strong prior belief that the hypothesis you're testing is false.

>> Or in other words, you have a strong prior belief that the hypothesis you're testing is false.

Yes, that is true for the dice roll; You already know most die aren't rigged.

I don't see how this affects the results of optimizely's product; You do not have a strong prior belief if the hypothesis is true or false.

Also note that the number of observations in the article used were in the thousands.

The p-value means "chance of this happening, if the two were equally effective". So 1-p means "change of this NOT happening, if the two were equally effective".

Note that in frequentist statistics, the sentence "Probability this is all luck" doesn't even make sense. Either the universe is so that it was luck, or it is not. In frequentist statistics you do not assign probabilities to models of the universe. You only assign probabilities to observations, given a fixed model of the universe.

I think you'll find this is a problem with all A/B testing apps out there. I haven't seen a Bayesian A/B testing app.

Reply on news.ycombinator.com