Ask HN: Is anyone using machine learning for A/B tests? | Hacker News Reader

Ask HN: Is anyone using machine learning for A/B tests?

4 pointspmtarantino7 comments

I am starting to apply machine learning in different ways, and I would like to know if someone is using machine learning for A/B tests.

For example, is is saving data and then showing A or B based on that data and improving it with each new case. I am not just talking about if A or B converted (that would be only success %), but more data like date, time, gender. For example:

I just came with the next idea: Maybe on the night, your visitors are tired for the day, so if you put something like "Register now. Only takes 2 minutes", has more conversions because has a better effect on tired people.

I am sorry if I am just divagating (and sorry for my English)

View on HN

There existed at least one project to do this, but you essentially can't justify it unless you are an ad network with hundreds of millions of impressions. The statistical confidence math is otherwise prohibitive. Ask if you need elaboration.

When you say "hundreds of millions of impressions" do you mean that literally or are you meaning "a lot of traffic"?

If it is the former then I would like some elaboration; my intuition is surprised that the amount of data required would be that large.

He is talking about Google. Probably Adwords but maybe other Google properties too.

I'm sure that Google and other large ad networks do this.

But I also see small/medium size ecommerce retailers doing "People like you also bought" type stuff. They don't do it as well as Amazon, but sometimes it seems relevant.

So I'm curious about whether hundred of millions of impressions are necessary for this type of optimisation (and that the good recommendations I've seen are just a coincidence - quite possible) or if saying "hundreds of millions" actually just means "quite a lot"

What makes you say that ? - assuming a reasonable number of variables I would think you'd only end up needing that order of magnitude if you split every combinations of variables into it's own test group to do A/B testing.

Presumably the reason that you'd apply a machine learning algorithm in this case is that you could extrapolate from a much smaller test group.

That type of business logic is actually quite straightforward. I was involved in a project many years ago, for which we got a patent I think, that involved detecting if time of day, distance from home and other factors could be attributes of ad serving.

Well, Myna https://mynaweb.com/ uses multi-armed bandit algorithms for A/B testing (or A/B/../N testing if you prefer).

An n-armed bandit problem is a type of reinforcement learning problem, and the associated algorithms for solving these problems are considered machine learning.

Reply on news.ycombinator.com