> And, quite frankly, the first victims of A/B testing are beauty, elegance, charm, and grace.
Right, and the first casualty of a designer's-opinion-only approach is "beautiful" designs that users hate. (Of course, neither premise is necessarily true, this is just a baseless appeal to emotion. Who could be against "beauty, elegance, charm, and grace"?) Can the author demonstrate from, say, ABtests.com or Whichtestwon.com that this is the case? Or did he simply have a bad experiences at highly data-driven companies? Is Netflix, for example, guilty of sacrificing "beauty, elegance, charm, and grace"?
> Instead we get a unsightly pastiche of uneven incrementalism lacking any kind of holistic cohesiveness or suggestive of a bold, vivid, nuanced vision that inspires users.
Such as? Is the author suggesting we go with poor-performing designs because in one or a few people's opinion, they "inspire" users? How do you know users are inspired if their performance suffers?
> It is the implicit charter of a high-quality design team (armed with user researchers and content strategists!) to propose something a user may not be able to imagine, that is significantly better, since they are so conditioned by mediocre design in the mainstream.
IMO, it's implicit that a high quality design team could come up with 2 or more such designs, or design variations, to test. Not only that, that they would come up with such design variations to test that are cohesive, elegant, and beautiful.
> A/B testing may only be as effective as the designs being tested, which may or may not be high quality solutions. Users are not always the best judge of high quality design. That’s why you hire expert designers of seasoned skills, experience, judgment, and yes the conviction to make a call as to what’s better overall.
Make a call based on what? Gut feel? The quality of designs being tested is not a problem of A/B testing -- if you only want users to choose from "high quality design", only test "high quality design". I don't understand the objection here. If a designer is truly an expert with seasoned skills, they should be the first to push for A/B testing of their best ideas, not retreating into vague notions of what they feel is best. There is no "One True Solution". Designers should own and drive the A/B testing process, not feel like they are victims of it.
> As is true with any usability test, you gotta question the motives behind the participants’ answers/reactions. Instead, biz/tech folks look at A/B test results as “the truth” rather than a data point to be debated. Healthy skepticism is alwayswarranted in any testing. Uncovering the rationale for a metric is vital.
People buy or they don't. They click or they go. You can't "question the motives" or hundreds or thousands of users, that doesn't make sense. You can't wish away the data.
> A/B testing is typically used for tightly focused comparisons of granular elements of an interface, resulting in poor pastiches with results drawn from different tests.
Poor A/B testing practice may be a problem, fine, but let's not throw out the baby with the bathwater.
> How do you A/B test novel interaction models, conceptual paradigms, visual styles (by the way, visuals & interactions have a two-way rapport, they inform each other, can’t separate them–see Mike Kruzeniski’s talks) which may vary wildly from before?
Uh, you run an A/B test and measure what you're doing.
> Would you A/B test the Wii or Dyson or Prius or iPhone? Against what???
Category error. We're talking about graphic/interaction/web design. All those companies absolutely should A/B test their web sites, for example.
> A/B testing locks you into just two comparative options, an exclusively binary (and thus limited) way of thinking. What about C or D or Z or some other alternatives? What if there are elements of A & B that could blend together to form another option? Avenues for generative design options are shut down by looking at only A and only B.
Seriously? Even basic free tools like Google Website Optimizer let you A/B/n or multivariate test.
> Finally A/B testing can undermine a strong, unified, cohesive design vision by just “picking what the user says”. A designer (and team) should have an opinion at the table and be willing to defend it, not simply cave into a simplistic math test for interfaces.
Sure, and not A/B testing can undermine a business by costing them thousands of dollars by not "picking what the user says". Pitting design teams against a "simplistic math test" is a silly way to polarize the issue. The designer or design team should be thrilled they have such a wealth of data at their disposal to come up with the best design possible. It's not about fighting, defending, or caving, it's about using the data to know -- and not blindly guess about -- what's going on.
> A/B test results perpetuate a falsely comforting myth that designs can be graded like a math test, in which there’s a single right answer.
The author asserts this several times throughout the piece but never provide any evidence. Will they shed some light on real case studies where this happens?
Indeed, if there is any "falsely comforting myth" doing the rounds, it's that designers can guess and know what thousands of disparate users will prefer within a single percentage point of accuracy. If you can guess that well, I'd like you to pick my stocks.
> And you risk dissuading top quality design talent from joining the team’s cause for good, beautiful, useful designs that improve the human condition.
Who's "human condition" are you improving by potentially harming client's businesses by blindly rolling out designs which may in fact cost them thousands (or hundreds of thousands, or millions) of dollars, and therefore jobs, and so on...? This is another silly, polarizing argument.
tl;dr: If you want "good, beautiful, useful designs", then A/B test "good, beautiful, useful designs". It's not us vs them. We can have our cake and eat it too!