A/B Testing is not Snake Oil.
visualwebsiteoptimizer.com
visualwebsiteoptimizer.com
I think the ideal design and development cycle incorporates A/B testing at the end to spot any outliers: You should already have done in-person usability and user testing. It's extremely cheap and, in some cases, almost magical. Seeing a real person use your product will guarantee you plenty of "a-ha!" moments.
After testing with real users, I think it's appropriate to test with statistical users: You've already tackled the most glaring issues with what you're building, and the knowledge of how real people use your website can often help show you the "why" of the data, rather than just "what".
Steve Krug's "Don't Make Me Think" has a chapter near the end on user testing on a shoestring budget. I'd highly recommend it.
You'd better be sure those "1 in a 100" results had a confidence level well above 99%. I think misunderstandings of statistical analysis play a large part in the mistrust some people (mistakenly) have in A/B testing. If you get your analysis wrong, you'll slowly realise that the promised gains of the test turn out to be lies, hence the comparison to snake oil.
I've already posted this link today, but I really recommend this guide: http://www.evanmiller.org/how-not-to-run-an-ab-test.html
Run enough tests, and you will get statistically significant but bogus results.
Many times, when we declare a winner in test and if you keep running it, you may see that eventually it does not perform as good.
But I agree my comment doesn't negate Jabbles' point.
If you run 100 tests, a small number of them will be FALSE positives, their confidence level will be above 95% even though they are, in fact, just statistical anomalies.
That's why it's 95% confidence and not 100% confidence.
What it means, by hiding all your failed tests, is that you are probably only writing about FALSE positives.
If you can't understand that, how can we trust any of your blog posts?
NB/update: Even though I'm generally pretty good at maths, I've always found statistics extremely hard. I totally understand how hard it is because it can produce such mind-boggling counter-intuitive results. But if you don't understand something that's key to your domain you should learn about it, not gloss over it.
Regardless of the statistics though, I personally think http://visualwebsiteoptimizer.com is very good.
However I have seen people here on HN make posts about A/B testing that even with my shallow understanding of statistics makes me raise my eyebrow that they really don't get it.
I do understand statistics behind A/B testing quite well but there are many subtleties (especially related to interaction effects) that I still need to understand and learn on the way.
It became apparent that Taguchi wasn't really appropriate or sufficient for web-based testing, so the team learned, devised, and implemented more appropriate MVT models.[2]
One notable bug that we discovered involved a self-optimizing test. The idea was that, once we reached a certain confidence level, we would slowly grow the number of targets that were fed the most successful variant.
We had a minor (on the order off off-by-one or switching a < and a <=) code error that grew the successful variant too quickly, at a point where the confidence level was effectively non-actionable.
As I recall, it took us about six months to notice, and none of our clients noticed.
MVT, and especially our implementation, is obviously much more complicated than straightforward A|B testing. Given the fact that no one was able to sniff out such an obvious error when their tests didn't improve conversion as much as expected has left me with the idea that, while testing is not snake oil, I have 99% confidence the population involved in split testing has only a superficial idea of what they're doing.
[1] I had previously implemented a very simple Apache plugin, mod_gating, that I should clean up and throw on github. Most of the work was in the lexer for the configuration file. :-)
[2] Much of the design of appropriate statistical models was done through consulting with statistics departments at a couple local top-ten universities. We figured advanced stats is like cryptography, if you're not an expert in the general field and you come up with a "proprietary" solution, you're probably screwing something up.