Hacker News Reader

Comment by pil0u | Hacker News Reader

What resources would you recommend to implement an A/B test evaluation framework using a Bayesian approach?

I would love to embrace (or at least try to) such a new approach, but it feels like without a PhD in stats it's hard to get it started.

View on HN

It really is not; Bayesian statistics is way more intuitive. For a practical guide see https://dataorigami.net/Probabilistic-Programming-and-Bayesi...

For Bayesian ML see https://probml.github.io/pml-book/book1.html and https://www.bishopbook.com/

For Bayesian statistics see http://www.stat.columbia.edu/~gelman/book/

Specifically for A/B or A/B/N testing, you can use a beta-bernoulli bandits, which give you confidence about which experience is best and will converge to an optimal experience faster than your standard hypothesis test. Challenges are that you have to frequently recompute which experience is best and thus, dynamically reallocate your traffic. They also only works on a single metric, so if your overall evaluation criterion isn’t just something like “clickthrough rate”, this type of testing becomes more difficult (if anyone else knows how multiple competing metrics are optimized with bandits, feel free to chime in).

Beta-Bernoulli multi-armed bandits (BB-MAB) are definitely a good way to get started on a Bayesian version of A/B testing where you've the additional benefit that your population is dynamically allocated to the most performant option (actually this dynamic allocation makes it similar to interim analysis [1] rather than vanilla A/B testing).

There are some caveats though - and I mention these from the experience of running such solutions on a large scale in production. First, BB-MAB can't adapt to context by design. They only look at click/no-click behavior across the population. So, if your population has two distinct segments - youth and elderly - who behave very differently wrt purchases, the BB-MAB won't pick a different winning advt. per group; its blind to these groups.

The solution is to use something like a contextual MAB - which assimilates user features (or whatever you might throw at it) into the MAB. There are simple ways to adapt simple MABs to the contextual setup [2] (in my experience, these can also be effective) but, of course, the literature in this area is wide and deep.

A second caveat is that if the ratio of the size of the pool of advts. to the number of impressions is high, the BB-MAB won't converge or converge to a good optima; the search space is simply too large relative to the data. In cases like this it becomes important to begin with the right Beta priors, instead of the standard recipe of starting with a Beta that looks like a uniform distribution.

[1] https://en.wikipedia.org/wiki/Interim_analysis

[2] https://arxiv.org/abs/1811.04383

I am not the expert and humbled by all the experts in this thread.

But I wanted to say that when I looked A/B testing few years ago I started off with this book.

https://github.com/CamDavidsonPilon/Probabilistic-Programmin...

And somehere down the line he will introduce the Beta/Binomial method for A/B testing.

In my (again humble) understanding the benefit of doing this in the Bayesian way is both that you can actually get an understandable answer, but also that the answer does not have to be the final result you can continue by adding a loss function for example.

Statistical rethinking by McElreath is an exceptional resource for Bayesian modeling. It's very accessible and gives you some good examples to work through.

Some very basic Bayesian models can go a long way towards making informed decisions for a/b tests

Here's a recent webinar from PyMC Labs' Max Kochurov (with some slides and code too):

https://www.youtube.com/watch?v=QllfKQH-yQ4

One straightforward approach is "Test & Roll: Profit-Maximizing A/B Tests" https://arxiv.org/abs/1811.00457

Reply on news.ycombinator.com