Why you should Stop using A/B Tests

Popularizing testing methodologies is a good subject, but you're way off base in terms of what A/B testing is and how it should be used. Hopefully you're just going for a dramatic point of view to drive responses, but as a point of professional courtesy, that's ok in your lead, not your actual content.

1. AB is not about determining the effect of various factors in isolation. Its about quickly reaching a solution that is close to optimal in situations where you can't feasibly generate the kind of data you'd need to build a model.

2. AB is not poorly suited to dealing with "externalities," confounding variables, etc. It is the only method that is suited to dealing with them.

3. This does happen, but it is more of a failure in the selection of WHAT to test, than it is a failure of the methodology itself. An understanding of optimization strategies is helpful in avoiding this trap. In practice I think what most people are doing is closer to simulated annealing than greedy anyway.

Ping me @TC_Davis on twitter if you want pointers to resources on any of the above.