Not many people have enough traffic to A/B test small effects and reach significance without running the test for multiple years.
I don't use CUPED in my tests... how much can it reduce wait times?
Not many people have enough traffic to A/B test small effects and reach significance without running the test for multiple years.
I don't use CUPED in my tests... how much can it reduce wait times?
What's really important if you want to improve a website via A/B testing is a constant stream of new hypotheses (i.e. new variants). You can call tests "early" so long as you have new tests lined up it boils don't to a classic exploitation/exploration problem. In fact, in early development rapid iteration often yields superior results to waiting for significance.
As a website matures and reaches closer to some theoretical optimal conversion point, then it starts becoming increasing important to wait until you are very certain of an improvement. But if you're just starting A/B testing, more iteration will yield greater success than more certainty.
Another way to say that is: you can randomly pick a winner
Taking a long time to reach "significance" just means there is a small difference between the two variants, so it's better to just choose one and the try the next challenger which might have a larger difference.
In the early stages of running A/B tests being 90% certain that one variant is superior is perfectly fine so long as you have another challenger ready. Conversely, In the later stages of a mature website when you're searching for minor gains you probably want a much higher level of certainty that then standard 95%.
In either case thinking in terms of arbitrary significance thresholds doesn't make that much sense for A/B testing.
But for incremental, smaller changes, calling early is probably gambling.