Effective Web Experimentation as a Homo Narrans

One of the advantages of the bandit algorithms we favour is that you have to be explicit about your quality metric up-front. This can also be a disadvantage as it can be difficult to tradeoff different metrics, but on balance we find it helpful for our clients.

I think one of the main issues with A/B testing, which was also present in the article, is the over-reliance on p-values without considering effect size. The changes show were significant, but none of them had a large effect. The largest was a 9% increase ("Registered") but here we're talking about going from 0.46% to 0.42%, which is pretty much nothing to nothing.