Why A/B testing will loose against advanced AI, all the time

medium.com

5 pointsfreemachines3 comments

Lot of words, not a singe real use case. Do they have a working prototype? Is the AI better than humans?

The idea of A/B testing is that humans are not good judging which version is better, so the only solution is to run an experiment, so in many cases you will get better results with an A/B test.

And there are also variations where the split is not constant, like the mutiarmed bandit. If A/B test is better or worse than multiarmed bandit is also a good flamewar topic.

Regarding that last bit: You will clearly get better results by using A/B tests than by having humans take a guess as to what works best – but as you already pointed out, those are not the only two possibilities. Different varieties of mutli-armed/contextual/etc. bandits, and then the new wave of reinforcement learning tricks (i.e. AI) are much better suited to the problem.

(Full disclosure: I didn't write the article, but I am the CTO of the company that the article is about.)

I would argue that A/B testing is close to it's end for 99% of it's current business applications. Right now there is a very limitied range of AI products that aim to replace A/B testing, but that will change soon enough.

Reply on news.ycombinator.com