Thanks, I have added it to my reading list.

It could make it worse. Just look at Amazon's homepage.

I don't think so. It would certainly help in some specific cases and some ideas are certainly worth testing out. However : -A/B testing can only answer some very specific questions. Good UX is not just measured by engagement. -some of these decisions are just bad practices that are widely recognized : for example we were asked to add modal popups pretty much everywhere. -we are severely shorthanded and implementing these changes and the A/B tests can be a very significant time investment. Especially when they are abandoned before going live for the reasons given by everyone but the person in charge.

Think of the eternal vastness of configuration spaces (even in domains as restricted as, say, static HTML/CSS pages) and then you know why AB testing is not a solution to everything. (Multivariate testing, too. It’s just slightly more elegant in some respects, but suffers from the same basic deficits and additionally has its own issues.)

Time is limited and traffic is limited. You simply cannot test everything, in fact, the amount of variants you can actually test during, say, a year may be shockingly low compared to the amount of variants possible (at least if you are not Google – but most aren’t).

Without a strong sense of direction and discerning eyes and hands that provide plausible variants to test, AB testing can be quite useless. AB testing is a great tool to verify hypotheses (and those may well be competing hypotheses), but I really question its usefulness as a creative tool.

If you can plausibly only do a couple of AB tests a year (the possibility of running an AB test being a function of traffic and time) you better not waste that opportunity on button colors or some similar nonsense.

Sure, evolution through natural selection produced some awesome designs and is basically a completely blind process based purely on testing of different variants with only slight and random differences with no involvement of any designer at all (sound familiar?) – but it also took billions of years. Most designers just don’t have that kind of time …

To more directly answer your question, AB tests can certainly be used to defend against bad hypotheses. If there is a culture of testing and new designs have to be battle tested then that’s less of an opportunity for someone in a position of power to screw things up by fiat. However, that’s also a tremendous waste of time and efficiency if bad designs that could be sieved out quickly by simple heuristics have to also first be tested, which is just a slow process and a wasted opportunity to test something better.

Maybe think of AB testing as a last line of defense, but be careful, AB tests can be paralyzing. (Or rather, first AB tests paralyze everything, then the boss is annoyed by how slow everything is going and changes everything by fiat and immediately without any testing at all. From one extreme to the other …)

I don't think the reason is any need for research. There have been quite a few occasions when I've made suggestions regarding how to fix obvious, major flaws. I've stopped bothering, because the response in all cases has been - not "sorry, we don't have time to implement the fix," which would be entirely forgivable, but some polite variant of "no, this is our product not yours, so we'll do things the way we want, not the way you want, so get lost."

On the bright side, that means if you are willing to set ego aside, you gain an immediate and substantial advantage over the competition.

There's such a thing as just-in-time design, where you only put as much effort (money) into it as it deserved. All rom-upgrade utilities look like suck, because, well, you only use them once and they're free. And so on.

Well, of course not.