I just get the impression that someone who wants to do A/B testing after reading this article will just create a different version of the website, and probably analyze the average of the results in each version and end up deciding solely on that.

Hope they don't miss the second point of the do's, it should be more emphasized:

> Don’t conclude too early. There is a concept called “statistical confidence” that determines whether your test results are significant (that is, whether you should take the results seriously). It prevents you from reading too much into the results if you have only a few conversions or visitors for each variation. Most A/B testing tools report statistical confidence, but if you are testing manually, consider accounting for it with an online calculator.

By the way, Excel can do it.