Hacker News Reader

Comment by jdlshore | Hacker News Reader

@darrenix, I see that your sample sizes are pretty low, and your actual conversions are incredibly low. Are you controlling for statistical significance, and are you accounting for repeated significance testing errors? [1]

Genuinely curious, not trying to shoot down what looks like an interesting approach.

[1] http://www.evanmiller.org/how-not-to-run-an-ab-test.html

View on HN

The actual conversion numbers are higher because, as I noted in the article, I'm only reporting number for web forms in order to keep things simple. There's also email and phone inquiries to count so you can basically multiply by 3x.

As for sample size, we're vigilant to make sure that we reach statistical significance. Typically our tests will encompass about 2,000 data points per group.

Our job is made easier by the fact that we're looking for huge wins so when we see a 10% gain with a 99% confidence, that's still a "fail" scenario for us.

It's possible to work with these sample sizes using survival analysis, but I doubt Bayes would find the difference even with a really well initialized prior. I don't think that traditional significance testing is the answer here, those require even larger sample sizes (b/c you're not limiting the space of the estimate). It's all about limiting the domain-space. You know conversion rates are never over 10% on your homepage, why would you not exclude that to give your statistical work more power? That's at least how I think about it.

It sounds like your stat-fu is way better than mine. :-) Is there a layman-friendly article you can point to that explains what you're talking about in more detail?

Can you elaborate? I'm not familiar with survival analysis for AB testing

Reply on news.ycombinator.com