False premises common in anti-A/B-testing arguments | Hacker News Reader

False premises common in anti-A/B-testing arguments

medium.com

34 pointsnoelwelsh17 comments

Some of these fall premises, are actually admitted to be true in the article... wtf?

Also, a lot of the quotes are not so much against a/b test, but people who claim a/b testing is a panacea, and are recommending the wrong tools to the wrong people. (ie tools based around static content to people with entirely dynamically generated content)

basically the biggest problem with a/b testing is that some of it's most noisy proponents have no idea what they're talking about. (this is true of a lot of things)

The biggest problem I have with A/B or Split testing is that it makes it easy for inexperienced people to assume there are multiple right answers to every problem and that they all warrant 'discussion' or testing... Most often this is not the case, and people who jump straight to A/B everything are the sort who don't really have/trust any experience and would rather spin wheels and throw spaghetti at the wall.

That said, sometimes there are two (or more) options that warrant testing (say a marketing strategy) and then it works great.

Test strategy, don't test everything, don't test design principles.

The biggest problem I have with not A/B testing is that it breeds a culture where people assume they really know what's going on.

Common sense is often only common sense in retrospect!

I've always thought the problem with A/B testing is that it's mostly hill-climbing optimization which is likely to hit and get stuck at a local optimum.

You could A/B test two completely different business models if you had the resources to do so. Your imagination and your bank account are the meaningful limits on what you can test, not the technology.

That works if you've narrowed it down (via some other process) to two choices of designs. A common argument is that A/B testing should replace unscientific, subjective design processes with data-driven design. But doing that basically reduces to applying randomized-optimization methods (genetic algorithms or whatever your favorite is) to everything: you need a way of mutating, testing, and repeating that explores the entire design space, or at least some relevant portion. And that tends to be difficult to implement in a way that works in complex domains where you haven't formalized all the elements.

A/B testing does work well for: 1) choosing between a small number of pre-selected options; or 2) exploring local variations in the neighborhood of an option. #1 is a very traditional kind of experimental study where participants are randomly assigned to two buckets, and #2 is a setting where randomized optimization tends to work well.

Agreed completely. To put it more simply: don't use AB testing to actually create your site by throwing mud on the wall. First design a good site, and then use AB testing to slowly tweak it to the closest best choices.

Just to add: unless you are testing google.com or some other top 10 site, #2 is not going to work well as each bucket will be too small to complete the AB test properly. eg. 3/100 users in bucket 92 converting is no proof that it is better than bucket 47 with 2/100 users converting.

I'm agreed you can't test everything -- but that's a limitation imposed by scarce resources (developer time, money, experimental subjects), not by the technology per se. And I'm agreed that trying to enumerate the space of possible designs is a mostly futile effort for most businesses.

However, if we take a broader view you can see a market as an experiment. Different companies in a competitive space (e.g. MP3 players before the iPod's dominance, A/B testing software now) choose particular feature sets, price points, and so on. Some are successful, and some aren't. It's difficult to learn from these experiments what contributes to success, as so many factors are mingled together and there is no systematic exploration of the space. As such, a common response is to assign success to the intangible -- "vision" or "genius" -- but I think this is taking the easy way out. Larger companies certainly use the market as a system o judge companies to acquire, and then often get burned when they fail to understand and maintain what brought about the acquired company's success.

I agree with the first point, but maybe it's my AI background: "that's a limitation imposed by scarce resources" is a pretty major limitation! If I had infinite time on infinitely fast CPUs with infinite human input, genetic programming algorithms writing code could completely replace human programmers and software architects. The reason they can't is because of a lack of infinite resources.

He addresses exactly that point with False Premise #2. It really depends on how you use A/B testing, if you only do it to test small incremental changes then yes, you're correct, but there's nothing stopping you from testing two totally different features/designs with A/B testing.

To which I'd respond that from personal experience, on a large corporate project, it's difficult to get authorization to do just that.

And even so, testing just two different designs is rolling the dice rather than exploring feature/design space in search of a new optimum.

As examples, I personally hate what's happened to the android clients for GMail and Google Maps. The former seems a victim of incremental change while the latter seems the consequence of a radical redesign that hit me out of the blue in the middle of trying to drive somewhere when I was already late.

"And even so, testing just two different designs is rolling the dice rather than exploring feature/design space in search of a new optimum."

A/B testing doesn't force you to innovate through radical design changes — you can use it just as easily for incremental improvements.

Only if you keep testing variations on the same thing. There's nothing about A/B testing that says (or drives you toward) doing that, per se.

A/B testing doesn't necessarily mean "test two subtle shades of the same thing." For a great example, see the Highrise A/B test: http://37signals.com/svn/posts/2991-behind-the-scenes-ab-tes...

This is a good example that disproves many of the false premises mentioned in the OP, as the clear winner of the A/B tests was a bold dynamic design with lots of character, that was completely different from the original designs.

Great example to bring into the conversation. One complexity to consider is that the long form versus the smiling person may have retained completely different user demographic. To me, the smiling person may represents an easy, idiot proof service with customer support, but may be lacking more advanced technical features, which is fine if you are set up to cater to that demo. Technical services like Github may suffer from increased sign ups from the wrong demo if it means their forums are swamped with beginner level questions that they aren't equipped to hand hold.

This would be 100x more useful if it was "Avoiding common pitfalls of A/B testing" -- since the author admits that poor usage of A/B testing makes most of the premises true.

His point is that these outcomes aren't inherent in the process of A/B testing, not that they aren't consequences of certain types of A/B testing.

Reply on news.ycombinator.com