Comment by jaggederest | Hacker News Reader

Comment by jaggederest | Hacker News Reader

Parent Full thread

> Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through.

It's almost guaranteed to ensure only false positive work makes its way through. If you're picking 0.05 as your P value, and you're running dozens to hundreds of tests, your false positives are almost certain to exceed your actual positives.

When I'm working for clients that do a lot of A/B testing, I suggest that they should always run A/A tests to ensure that they're not incorrectly rejecting the null hypothesis. If your A/A tests are showing significant differences, you have a problem in your testing pipeline that by definition can't be cured by more testing. You need holdout groups and selectivity about what to test, instead of just throwing everything at the proverbial wall.

Even checking A/A tests won't surface all the issues. A proper A/B test is one that samples over a long enough time to adjust to the true audience of the service.

For example, imagine a costume shop that ran a couple dozen A/B tests over the summer. Those results may look statistically significant. They may even stand up against the A/A test. But people that buy costumes in the summer are very, very different than people that buy them in October, and if 90% of the store's business is in the run up to halloween, then all these micro optimizations could actually make your total business performance worse.

I'm a A/B testing skeptic too, though I admit they have a time and a place. My favourite are ones that can be reasoned about as actual hypotheses. This usually involves some degree of data analysis or segmentation. For example, increasing font sizes may boost conversion, and a later analysis shows that this was almost solely a lift in conversion rates amongst the 45+ cohort. The data in this case isn't just blindly driving design decisions, it's helping inform the staff on how to better design in the future for the audience we have.

Well, if you are running hundreds of tests with 0.05 p-value, you will get plenty of false ok A/A tests, and there isn't much of a reason to expect them to be correlated to actual signal on your A/B tests.

A/A tests do test your methodology as you said. But they do not fix a p-value one order of magnitude higher than it should be. (And yeah, I'm aware you know that, but your comment places them on the same context, so it got misleading.)

Great insight. Without this approach, A/B testing could be used to generate an infinite stream of meaningless work

The first company I worked for, and also the first company I saw A/B tests at, once ran an A/A test because someone was a bit skeptical about some of the test results that had been claimed.

Predictably, whatever metric we were watching on it (probably conversion) swung wildly to either side over the first few days. The look on some of the product managers' faces was pretty great. After about 2 weeks, it settled into a steady state where each "version" performed equally (measured cumulatively, so just large numbers in action).

The conclusion from this exercise was...

"It takes 2 weeks."

¯\_(ツ)_/¯

That's why we calculate stuff like effect size and power of a test (or even better, use Bayesian statistics); just p < 0.05 is practically almost meaningless.

Reply on news.ycombinator.com