This is exactly the problem with people implementing split testing and they're hurting their business.

They hear about it, run off and implement the a/b versions of their pages and add some tracking. Proceeding then to draw incorrect conclusions and make bad decisions based on them. They don't go read up on it enough to understand the statistics behind it. I used my stats book from Uni (I knew it had a better use than a laptop stand, a java book now has that honour) but there are plenty of tutorials out there. For example: http://visualwebsiteoptimizer.com/split-testing-blog/what-yo...

It really is a toss up between statistics and medicine for which is the most misunderstood of the sciences.