I've long suspected that A/B products "confidence" stats are intentionally bad. I see far too many egregious declarations of "wins" in VWO. Even if they temper it by saying "you should wait a full seven days", I still think they do a large disservice by not forcing larger sample sizes before steering you in any direction. Those tools make it very easy to run tests and assuming any of their users are sophisticated enough to make their own statistical conclusions is such an obvious mistake that they have to be aware of it.

The conversion optimization consulting companies I've seen aren't much better. They have no problem reporting "increased" revenue and rates well before there is enough data to support that. I'm calling these types of tests A/B Dog and Pony shows. It's easy to report these gains up, get some short-term love from your boss or client, and hope the chickens never come home to roost or things are forgotten before they do.

There was a blog post on Unbounce last week talking about a "successful" A/B test where the combined conversions of A and B over the testing period was 12. This is the kind of information a company specializing in split testing landing pages is providing. The article was since taken down, but it's pretty clear that a large subset of conversion rate folks and the products used for conversion rate optimizations have little interest in dealing with legitimate results.

Spend some time reading A/B testing article on Inbound.org or GrowthHackers. I'll call them mostly a 50/50 mix of articles saying why your A/B testing results suck and don't stick and articles bragging about A/B testing results that aren't statistically valid.