A/B Test Significance: Test Outcomes for Statistical Significance

Well it's not completely wrong, but it does assume a power of 50%, which is not a good default.

Power is the probability that your experiment (A/B test in this case) will detect a difference if it truly exists. This is different to statistical significance which is related to the probability that you will conclude that A and B are different when no difference truly exists. Look up type I and type II errors if you are interested in this.

So if you use the sample size suggested by this website you will only have a 50% chance of detecting that A and B are different (given that A and B are actually generating different response rates). This means you will miss the effect that a better page design has on your response rate 50% of the time. In clinical trials it is usual to use 80% or 90% power.

You can use our power calculator at http://www.sealedenvelope.com/power/binary-superiority/ to get the sample size for an A/B test with a more realistic power. The code is open source at https://github.com/SealedEnvelope/sample_size_calculators.js so it would be good if this website was updated to use a better default for power (or make it user selectable).

Realize that this is missing a key metric. Well, a lot of new teams running A/B tests do too. People focus too much on conversion statistics and forget about the actual sales numbers (as in money). Some setups will test worse (as in less total sales (in number of tickets)), but will have a higher amount of profit per sale. Making it more profitable with less sales. Some might say that then the correct test was that one. Err, no. You still need to keep testing for the conversions, but also need to pay attention to average transaction amount per test. Conversions are meaningless if you are only converting cheap customers who opt for the cheapest offering (by the way, raise your prices).

This is completely wrong. You need to specify power of the test. You are trying to detect if the new population mean Conv rate is higher (given a, say, 10% increase is economically meaningful to you). Otherwise this tells me nothing

Wow this is pretty cool! I've never seen anything like this before. And I'm a fan of the Boostrap too