Comment by btilly | Hacker News Reader

Comment by btilly | Hacker News Reader

I agree as well!

Here is what probability theory teaches us. The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs through Bayes' theorem. The challenge is how to best communicate this result to people who may have had a wide range of prior beliefs.

p-values capture a degree of surprise in the result. Naively, a surprising result should catch our attention and cause us to rethink things. This is not a valid statistical procedure, but it IS how we naively think. And the substitution of a complex question for a simpler one is exactly how our brains are set up to handle complex questions about our environment. (I'm currently working through Thinking Fast and Slow which has a lot to say about this.)

Simple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different.

Both approaches have a fundamental mistake. The mistake is that we are taking a data set and asking what it TELLS us about the world. When in probability theory the real role of data is how to UPDATE our views about the world.

This is why I have come to believe that for simple A/B testing, thinking about p-values is a mistake. The only three pieces of information that you need is how much data you are willing to collect, how much you have collected, and how big the performance difference is. Stop either when you have hit the maximum amount of data you're willing to throw at the test, or when the difference exceeds the square root of that maximum amount. This is about as good as any simple rule can do.

If you try to be clever with p-values you will generally wind up putting yourself at saving yourself some effort in return for a small risk per test of making very bad mistakes. Accepting small risk per test over many tests for a long time puts you at high odds of eventually making a catastrophic mistake. This is a very bad tradeoff.

I've personally seen a bad A/B test with a low p-value rolled out that produced a 15% loss in business for a company whose revenues were in the tens of millions annually. It. Was. Not. Pretty. (The problem was eventually found and fixed..a year later and after considerable turnover among the executive team.)

Simple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different.

I don't think this is a completely accurate portrayal of Bayesian stats. In Bayesian stats, there is no "real prior". Probability distributions are all subjective representations of belief. The prior is just what you believe prior to evidence, and the posterior is what you believe after you've taken evidence into account.

That said, moving away from p-values and towards something more robust is something the A/B testing industry needs. (Obviously I have my own opinion of what that something should be, and it's a bit different from what you are advocating.) There are far too many consultancies and agencies p-hacking their way to positive results ("hey unsophisticated client - guess what I made your conversion rate go up 25%!") and I'd love to see every one of them die.

There are different approaches a Bayesian might take. The one that I described is certainly among them, though it is not the only one.

I think the word "naive" is problematic here. Have you seen instances where Bayesians choose a prior that isn't at least somewhat informed by exploratory analysis?

A uniform prior would be one example. That's just saying "I've got no idea what the value is".

https://en.wikipedia.org/wiki/Prior_probability#Uninformativ...

There is a notion of an uninformative prior [0].

What would you do if you were studying a process that had not been previously studied, so you had no previous experience/data on which to base your a prior? If you wait, and choose a prior based on the data you observe then you are using an empirical Bayes method [1].

[0]: https://en.wikipedia.org/wiki/Prior_probability#Uninformativ...

[1]: https://en.wikipedia.org/wiki/Empirical_Bayes_method

It's quite common to choose a conjugate prior, which aids computation, and which is readily comprehensible within the discipline ("...and a Wishart prior for the covariance, of course..."). But which is, in effect, not at all informed by the data.

It's also common to have a complex, multi-level model setup which has a few hyperparameters (say, gamma distribution shapes) which are set basically arbitrarily. The idea being that the posteriors for the lowest levels (closest to data) will be learned/fitted, but the hyper parameters at the top-level of the model are fixed. This is very common in spatial statistics.

It's also common to make lots of (conditional) independence assumptions, just because they are convenient. Such as diagonal covariances, or conditional independence between "separate" elements of a model. But these independence assumptions are often just convenient or based on intuition, but not re-checked. Or, if they are checked and found wanting, it's "left for future work".

These practices are defensible, and attackable. But choosing convenient priors out of a bag of standard priors, without reference to the problem at hand, is very common.

EDIT: I forgot about the prevalence of uninformed priors. Thanks HN!

There is nothing more robust than p-values. The problem is that people call things "p-value of this experiment" when they aren't (for example, they may actually be talking about the "p-value of this other experiment" [which they possibly didn't even perform]).

Thanks for the very interesting thoughts.

Could you explain a bit how your rule of thumb works and why it's better than p-values? Why is the difference vs. the square root of the max available sample size a meaningful measure?

The idea is that you will decide when you've either expended as much energy as you are willing to, or when you're convinced that you won't make a different decision.

There is a simple symmetry argument that shows that the odds of a random walk reaching sqrt(N) in one direction and then getting to the opposite direction by N observations is exactly the same as the odds of a random walk reaching 2 sqrt(N) by the time you reach N observations. So the sqrt(N) rule is a simplified version of, "Call off the test when you've reached 95% certainty that we have our final answer."

You can easily make a million minor improvements. But I can't come up with a simpler rule meeting the business needs which is less likely to lead to dangerous misunderstandings.

"Decide when you pass this threshold, or when you've put out as much effort as you're willing to."

And the effort required to testing are easily understood as well.

"The cost of a bad A/B test is at most 100 lost sales and remembering it is there for 2 months." (This would be for an organization that expects 10,000 sales in 2 months, and is willing to leave tests up for that long.)

And if memory serves me (I'd have to redo the calculation) the rule of thumb in this case is that wins below 1% are called randomly, above 2% are usually called right, you're confident of getting the right answer above 5%, and even over many tests you are confident of never making a 10% mistake.

That last rule of thumb is important. Small organizations can't benefit from A/B tests. Large ones are stupid not to use it heavily. And very few people have a good sense of whether it makes sense for them.

Thanks! That's very helpful.

Did you know that your rule of "do the tests until significance is reached" denies the use of chain probability and most tests based on it?

You can show that in the limit, your rule will always reach any given threshold just by pure luck.

You really need to set the number of trials or samples before performing them.

Here's a more in depth explanation: http://www.evanmiller.org/how-not-to-run-an-ab-test.html

The procedure I suggested starts with setting the maximum number of conversions before beginning the test. It is emphatically not the procedure that Evan Miller criticizes.

In fact see http://www.evanmiller.org/sequential-ab-testing.html for Evan Miller suggesting a similar procedure to mine, based on past conversations that I have had with him. The difference between that one and mine is that I focus on "make the best decision available, even though it may be only little better than a coin flip" while he tries to only decide when you can decide with confidence. In past conversations we have agreed that each other's procedures are reasonable, but they achieve different goals.

The way I see p-values is a mathematical way to share how certain you are that something is in the realm of reality (not truth). It is not a formal conclusion of the results, and is nothing more than a "spoiler" indicating what you might also conclude.

This point of view is not supported by the mathematics. If you think otherwise, then you do not understand the mathematics.

Reply on news.ycombinator.com