A/A Testing

One can commit to a performance metric threshold prior to A/B testing. For example, a rule that states "New, nonmaintenance features must increase revenue for the product by at least 1.0%".

An objective rule like that neatly resolves the problem of biasing analysis after the test, and doesn't have to necessarily involve any complicated statistics.

Our favorite test is A, A, B, B

It’s surprising how often a test could be set up incorrectly or have other technical issues which this test can catch.

We try to ensure a statistically significant result on both A’s or both B’s before determining a winner.

It does take longer to run this test, but should give a much higher degree of accuracy.