Hacker News Reader

Comment by btilly | Hacker News Reader

Meh, you make it sound too hard. Here is a perfectly adequate A/B testing regime.

Assign visitors into buckets randomly. Track conversion counts only. Declare the winner of an A/B test whenever one gets 100 conversions ahead, or whichever is ahead after 10,000 conversions. Do this with every test. As long as there are no obvious interaction effects, you can run multiple tests in parallel.

If you wish, you can replace 10,000 with N where N is however many conversions you expect to get in a month. Replace 100 by the square root of N. Real differences below 2/sqrt(N) are basically coin flips. Even after many tests, you are very unlikely to have ever made an error as big as 5/sqrt(N).

If you want to do something more sophisticated than this, you need competent people. But this is worlds better than what most companies do. (And yes, it is better than what Optimizely does for you by default.)

View on HN

I appreciate what you're trying to say here, but you've basically just described the prototypical way not to run an A/B test. Ending a test as soon as you reach significance ensures a high rate of false positives, and doesn't clearly tell you when to end the experiment if there is no effect.

You have just pattern-matched to a response without understanding. And you're wrong.

See http://www.evanmiller.org/sequential-ab-testing.html for a very similar methodology presented by the very person you are citing. The difference is that I am aiming to always produce an answer from the test, even if it is a statistical coin flip, while he aims to only produce an answer roughly 5% of the time if there is no difference.

Maybe I am not understanding your post, but aren't you just declaring a winner after N trials, even if that is not significant? That seems to be a critical distinction here.

Right. We are trying to make a business decision, not conduct science.

If you go with whichever is ahead, you'll reliably produce correct answers for wins at that you can't reliably measure. Yes, there will be mistakes, but the mistakes are guaranteed to be pretty small.

If you insist on a higher standard, you'll learn which choices you were confident of..but now need to figure out what to do when the statistics didn't give you a final answer.

I think that the first choice is more useful. Evan prefers to clearly distinguish your coin flips from solid results. But in the end you have to make a decision, and it isn't material what decision you make if the conversion rates are close.

I'm sorry, I wouldn't call that framework A/B testing also much as running two experiments and seeing who's ahead after some arbitrary period of time.

I think this post summarizes it really well: http://bjk5.com/post/12829339471/ab-testing-still-works-sarc...

If you have any sort of decent dashboarding the cost of a wrong decision is really not all that bad compared to the cost of being a purist.

Oh boy. Let's replace statistics with guesses off of graphs!

That visible "zone of stupidity" is based on how long it takes to make a conversion which has everything to do with how your product works, and nothing to do with how statistics works. There is absolutely no difference between the graph you expect that leads to an accidental wrong decision and one that detects a correct difference - the patterns that you think you see don't mean what your brain will decide they do.

And more importantly, if you stop in 1/4 of the time that you would have been willing to run the test, the potential loss when you are wrong is twice the worst errors that you could make if you put more effort into it.

Have you ever been in an organization that rolled out an innocuous looking change that killed 15% of the business? I have. Over the 10 months it took to find the offending subject line, there was, shall we say, "significant turnover in the executive team".

Math exists for a reason. Either learn it, or believe people who have learned it. Don't substitute pretty pictures that you don't understand, then call it an explanation.

I admit I don't know the math well so I am curious to know how to fix my intuition:

Let's say you want to figure out the unknown bias on two coins. You flip both continuously and plot the percentage of heads you see. Due to law of large numbers these percentages will eventually converge to true probabilities (which is how I am interpreting the graphs in that blog post).

The bad case is if the two coins are actually "flatlined" in the wrong order so as a pattern matching human you mistakenly believe the rates have converged prematurely. I don't know how to work out the math on this but let's say a "flatline" is visually 100 points or so with no significant slope. Then this should be pretty rare right?

Don't try to do this by visual pattern recognition. Do math. There are plenty of statistical tests that you can use, use them. Any of them is better than looking at a graph and guessing from the shape.

If you want to try to understand what is going on, learn the Central Limit Theorem. That will let you know how fast the convergence is to the laws of large numbers. (There are two, the strong and the weak.)

Well, that is what A/B testing is. With a decision rule to try to make that determination in some sensible.

Usually the decision rule is stated in terms of a statistical confidence test. Usually the confidence test is done poorly enough that it doesn't mean what people think it does.

And the stopping procedure isn't actually very arbitrary. You choose N based on the most data that you're willing to collect for this experiment. And stop when you're confident about which version will be ahead at that point.

So this procedure leaves you confident of having the best answer that you will get from the most extensive test that your organization is willing to commit to running. And the cost to the organization of running it is capped at sqrt(N) lost conversions.

I take it you're the person who wrote the original to which your linked article responds?

Yes.

It took a lot of work to get down to the super simple version. :-)

You're describing the math of A/B testing, but not the work of analyzing data to decide what experiment to do next, designing the experiment, building and integrating it, then assigning visitors to the buckets and so forth.

The technical end of building / integrating / assigning visitors and so on tends to be relatively simple if it is done by competent people who know how to not overcomplicate things.

Coming up with ideas to test tends to be fairly easy if you've got a competent product person.

Perhaps I'm biased. I've built multiple A/B testing systems for multiple companies, and know what to do. However it didn't seem hard for me the first time either.

This seems like a nice application of two related principles: 1) The harder it is to tell which choice is better, the less it matters which you choose, and 2) a decision has value even in the absence of proof that it's the very best decision because it frees up resources for the next decision, which may be more important.

Depending on the implementation of "assign visitors into buckets randomly", this should include a test to ensure the populations are correctly randomized. "Unsophisticated" users of these heuristics are prone to make terrible decisions if the treatment populations are not properly randomized (this can be due to systemic bias caused by e.g. caching, or treatment effects like survival bias).

Can you explain the statistics behind this methodology?

Essentially the same as http://www.evanmiller.org/sequential-ab-testing.html except that he aims to not produce an answer most of the time by chance, while I aim to always produce an answer even if it is statistically a coin flip. (Because, after all, you need to actually choose a headline in the end...)

Reply on news.ycombinator.com