Comment by jfarmer | Hacker News Reader

Comment by jfarmer | Hacker News Reader

Parent Full thread

Every time you do a significance calculation and decide whether to stop the test or continue you increase the likelihood of a type I error (i.e., false positive).

So, your 99.8% confidence isn't 99.8%.

There are a few ways to compensate for this. The easiest is to fix your sample size and not reach any conclusions until you have tested that many users.

You can determine your sample size by, say, figuring out the minimum number of people needed before there's a 95% chance you observe a 10% effect size.

This is a problem in clinical trials where ethical questions arise. If the test appears harmful, should we stop? If it appears beneficial, isn't it unethical to deprive the control group of the treatment?

Anyhow, if you want to observe the results continuously, you need to use a technique like alpha spending, sequential experimental design, or Bayesean experimental design.

TL;DR: If you're periodically looking at your A/B testing results and deciding whether to continue or stop, you're doing it wrong and your significance level is lower than you think it is.

And this, ladies and gentlemen, is why we use Bayesian likelihood ratios. You can use whatever stopping algorithm you like on likelihood ratios and they will still be valid.

"Statistical significance", not so much. A simple test program I wrote showed that with a limit of 500 coinflips, flipping a fair coin will produce, on at least one out of the steps, a dataset which rejects the null hypothesis with p < .05, fully 30% of the time.

http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...

Right, but even frequentist methods can prevent an erroneous rejection of the null, if used correctly.

The likelihood ratio is just true no matter how you use it.

The meaning of an experimental result should not depend on the experimenter's private state of mind (such as the stopping rule they followed) if that state of mind does not affect the experimental apparatus. The likelihood ratio is objective, "statistical significance" is not. See link.

Totally agreed. But assuming that you wanted to calculate a posterior, what would you have chosen for a prior here?

(Note: Second paragraph in grandparent was edit, original reply in parent may have been to first paragraph only.)

Well, keeping in mind that experimenters should just report likelihood ratios, I'd have estimated a pretty low prior probability that one of two identical pages produced more signups than the other.

But I suppose that after seeing a likelihood ratio of 1000/1 favoring the hypothesis "each of these pages has a separate fixed conversion rate" over "both of these pages have the same fixed conversion rate", using Laplace's Rule of Succession (i.e. a flat prior between 0 and 1 for the conversion rate) in all cases, I'd think that something was more likely than not going on.

For academic journals, I think that what you propose is totally reasonable. I'll try to give a small taste for those not familiar:

For a new HIV test, I want to know the likelihood ratios, not other parameters which tend to be driven by the background HIV prevalence in the researcher's specific catchment area. My catchment area's HIV prevalence will differ, so other aspects of the test that are dominated by prevalence will not be the same in my population. However, the likelihood ratio will be.

The point that I didn't finish making before getting kicked out of the coffee shop was that, at some point, people will want to compute a posterior, so they'll need to figure out a prior. Your suggestion above is helpful.

Edit - Also, you're right in that I saw just the 1st paragraph when I initially replied.

Just to clarify this entirely correct analysis: It's not a "peeking" error, it's a "stopping" error.

As long as you decide on a minimum number of trials to make a decision, it's fine to look at the data as it is accumulating.

This is one of the more insightful comments I've read lately. I thought just upvoting wasn't enough for this one, so I wanted to say thanks and I'm going to refine my next split tests along your guidelines.

It's also worth pointing out that you can safely peek and make an early decision if you insist on a higher confidence interval for early stops.

The exact numbers depend on how often you test the confidence interval, but a multiple of 5-10 is for early measures will usually give reasonable results.

E.g., let's say you ultimately want a 98% confidence interval (2% chance of error). A 10x improvement is a .2% change of error, or 99.8%. Depending on how often you are sampling, an early result with 99.8% confidence can be equally accurate as a later results with 98% confidence.

Thanks for that, that's something I haven't seen anywhere. Do you have any references that explain why this error occurs?

When you take multiple peeks at the data as it begins to accumulate, you're essentially performing multiple testing.

Imagine that you wanted to see if A != B, with a 95% confidence. To oversimplify, this means that you're willing to accept that 1 in 20 times, you'll incorrectly reject the null and you will consider A different from B even though they are truly the same.

If you run 20 independent tests at once, each at a 95% confidence, then by chance you'd expect 1 to reject the null even if they're actually all null.

Now, if you repeatedly peek at the data as it accumulates for one test, you're doing something similar (not 100% the same because the tests aren't totally independent, but similar). To again oversimplify, you'd expect, by chance, to see "significant" results 1 in 20 of the times that you look at the data, even if there is no significant result. This is why you need to wait until you've collected all of the necessary data first, or implement a procedure to protect you from incorrectly seeing "significance" when it's not there. You might, for example, require a more stringent confidence level earlier on, and less stringent ones later.

Yep, that's what I realised from your and jfarmer's comments. It does make sense that you're sampling multiple times and peeking to see if you've reached that, thank you.

http://www.evanmiller.org/how-not-to-run-an-ab-test.html

http://news.ycombinator.com/item?id=1277004

Thank you.

Reply on news.ycombinator.com