Why designers don’t like A/B testing
ghostinthepixel.com
ghostinthepixel.com
> And, quite frankly, the first victims of A/B testing are beauty, elegance, charm, and grace.
Right, and the first casualty of a designer's-opinion-only approach is "beautiful" designs that users hate. (Of course, neither premise is necessarily true, this is just a baseless appeal to emotion. Who could be against "beauty, elegance, charm, and grace"?) Can the author demonstrate from, say, ABtests.com or Whichtestwon.com that this is the case? Or did he simply have a bad experiences at highly data-driven companies? Is Netflix, for example, guilty of sacrificing "beauty, elegance, charm, and grace"?
> Instead we get a unsightly pastiche of uneven incrementalism lacking any kind of holistic cohesiveness or suggestive of a bold, vivid, nuanced vision that inspires users.
Such as? Is the author suggesting we go with poor-performing designs because in one or a few people's opinion, they "inspire" users? How do you know users are inspired if their performance suffers?
> It is the implicit charter of a high-quality design team (armed with user researchers and content strategists!) to propose something a user may not be able to imagine, that is significantly better, since they are so conditioned by mediocre design in the mainstream.
IMO, it's implicit that a high quality design team could come up with 2 or more such designs, or design variations, to test. Not only that, that they would come up with such design variations to test that are cohesive, elegant, and beautiful.
> A/B testing may only be as effective as the designs being tested, which may or may not be high quality solutions. Users are not always the best judge of high quality design. That’s why you hire expert designers of seasoned skills, experience, judgment, and yes the conviction to make a call as to what’s better overall.
Make a call based on what? Gut feel? The quality of designs being tested is not a problem of A/B testing -- if you only want users to choose from "high quality design", only test "high quality design". I don't understand the objection here. If a designer is truly an expert with seasoned skills, they should be the first to push for A/B testing of their best ideas, not retreating into vague notions of what they feel is best. There is no "One True Solution". Designers should own and drive the A/B testing process, not feel like they are victims of it.
> As is true with any usability test, you gotta question the motives behind the participants’ answers/reactions. Instead, biz/tech folks look at A/B test results as “the truth” rather than a data point to be debated. Healthy skepticism is alwayswarranted in any testing. Uncovering the rationale for a metric is vital.
People buy or they don't. They click or they go. You can't "question the motives" or hundreds or thousands of users, that doesn't make sense. You can't wish away the data.
> A/B testing is typically used for tightly focused comparisons of granular elements of an interface, resulting in poor pastiches with results drawn from different tests.
Poor A/B testing practice may be a problem, fine, but let's not throw out the baby with the bathwater.
> How do you A/B test novel interaction models, conceptual paradigms, visual styles (by the way, visuals & interactions have a two-way rapport, they inform each other, can’t separate them–see Mike Kruzeniski’s talks) which may vary wildly from before?
Uh, you run an A/B test and measure what you're doing.
> Would you A/B test the Wii or Dyson or Prius or iPhone? Against what???
Category error. We're talking about graphic/interaction/web design. All those companies absolutely should A/B test their web sites, for example.
> A/B testing locks you into just two comparative options, an exclusively binary (and thus limited) way of thinking. What about C or D or Z or some other alternatives? What if there are elements of A & B that could blend together to form another option? Avenues for generative design options are shut down by looking at only A and only B.
Seriously? Even basic free tools like Google Website Optimizer let you A/B/n or multivariate test.
> Finally A/B testing can undermine a strong, unified, cohesive design vision by just “picking what the user says”. A designer (and team) should have an opinion at the table and be willing to defend it, not simply cave into a simplistic math test for interfaces.
Sure, and not A/B testing can undermine a business by costing them thousands of dollars by not "picking what the user says". Pitting design teams against a "simplistic math test" is a silly way to polarize the issue. The designer or design team should be thrilled they have such a wealth of data at their disposal to come up with the best design possible. It's not about fighting, defending, or caving, it's about using the data to know -- and not blindly guess about -- what's going on.
> A/B test results perpetuate a falsely comforting myth that designs can be graded like a math test, in which there’s a single right answer.
The author asserts this several times throughout the piece but never provide any evidence. Will they shed some light on real case studies where this happens?
Indeed, if there is any "falsely comforting myth" doing the rounds, it's that designers can guess and know what thousands of disparate users will prefer within a single percentage point of accuracy. If you can guess that well, I'd like you to pick my stocks.
> And you risk dissuading top quality design talent from joining the team’s cause for good, beautiful, useful designs that improve the human condition.
Who's "human condition" are you improving by potentially harming client's businesses by blindly rolling out designs which may in fact cost them thousands (or hundreds of thousands, or millions) of dollars, and therefore jobs, and so on...? This is another silly, polarizing argument.
tl;dr: If you want "good, beautiful, useful designs", then A/B test "good, beautiful, useful designs". It's not us vs them. We can have our cake and eat it too!
Except we weren't making games... We were making corporate intranets.
Examples: iPhone. No manual (probably there is now). It's unbelievable, really. But, you have to admit it works!
Anti-example: fire extinguisher. No tricky interface, please.
One reason I didn't see mentioned is user level. Beginners want/need simple. Later, they want more power. How do you keep the advanced features out of the way of the new users? Hide them! Advanced users will find them when they are ready.
You can't tell someone who really feels that narrative that, by the way, technical competence is not sufficient for success. (To say nothing of saying it isn't necessary.)
So why don't designers like A/B testing? Well, what's their narrative? User-loving highly-skilled artist scorned by people in positions of power. (Wow, we have a trend here.) They don't want to be told that a) users provably don't care that this design looks better or is more intuitive or b) that beautiful/responsive/intuitive/etc design is not always worth actual money.
Imagine this scenario: you bring four aliens to a table, you give them a deck of cards, and you explain the mechanics of poker (but not the actual rules). You get them to play, remain silent throughout the game, and at the end you announce the winner.
On the first game, an alien wins with a pair of sevens. So the aliens think "hey, it's the sevens that have value!", and pretty soon the aliens are fighting over the sevens, and they call it the game of sevens.
A|B testing gives you a tiny glimpse of a wicked system that happens to work under certain circumstances. I love A|B testing, but I know you can't make all encompassing announcements that state that buttons should always be beveled (as Jakob Nielsen says) because it depends on the specific circumstances of the design. Just look at gmail's new design, I'm sure it's pretty well tested and there's no trace of beveled buttons.
I remember an interesting piece you wrote on changing the logo for Bingo Card Creator, and you announced that logos are worth nothing. But then comes GAP with their horrendous logo and the internet goes batshit over it.
I think we are in the alchemy ages of design, and there is a lot of resistance in adopting scientific methods for validating our work. But we must understand that every test is a tiny glimpse of the overall rules of the game. Every little piece of evidence is precious in understanding the whole system.
The rules of cognition in regards to visual and interactive input may well be simple at the very low levels (neurons let's say), but even if we could isolate those rules the ability to cognitively extrapolate principles of design would be beyond the capability of the human brain.
I actually think it goes deeper than just narrative; there's a difference of ontology. To a designer, a "better" design means a more aesthetic one. To a businessman, it means a more profitable one. They're not completely contradictory, but they don't always match up either— so it shouldn't be surprising that there's some cognitive dissonance going on when the A/B test says the worse design is actually better.
A/B testing is at its best where the measurable quantity is short-term and largely independent of anything else, like conversion on a landing page. Things get more murky when you want longer-term things, like building a coherent/consistent brand, where every product/image/campaign you do doesn't wildly jump around--- imagine an Apple where every product was ruthlessly optimized in isolation, with no attention to whether it fit into Apple's larger design sense. A designer might not have the right answers there either, but the right thing to A/B test is a bit trickier, because you aren't testing decisions that are independent and easy to isolate. What you want is the overall set of decisions that maximizes 5-year profits, which might not be simply the sum of decisions that maximizes profit in every individual case.
One subset of solutions is to have someone responsible for vetoing elements of individual campaigns that might cause collateral damage bigger than the campaign's value. For example, a Disney landing page that wildly outconverts other possibilities might still get vetoed if it does something with the Disney characters that isn't in line with the image they want for them. Another is to have broader design parameters imposed ahead of time, with A/B testing restricted to a smaller set of design variables permitted to vary, like Apple does.
Deleted comment
A good example is WebVan. They blew a billion dollars trying to do internet-based grocery shopping at scale. They learned the world wasn't ready. They could have learned that for maybe 1% of that cost. And then had the other 99% of the money to find something that actually works.
I think my own startup is another good example. We started out with an idea that we thought was great. After funding, we discovered two funded competitors. We did a bunch of small-scale tests of our product, discovered it wouldn't scale, and killed it after 7 weeks. The competitors spent 9 months to a year and millions of dollars. You can see my co-founder describe our experience in this 5-minute Ignite talk: http://vimeo.com/24749599
Instincts are great. They're also frequently wrong. That's why we test our ideas vigorously before we bet big on them.
But for everyone who's championing split-testing, they should remember a little bit about where it first became popular: Direct Mail. Direct Mail copywriters/designers were the first people to truly use split-testing as a major part of their professional arsenal.
Split-testing gave us long-form sales letters with typewriter fonts, blue backgrounds, yellow highlighter effects, and pictures of Ferraris and McMansions and screenshots of commission checks.
Why? Because they all work better.
Don't get me wrong, I find those design elements you're talking about to be far from my tastes, but provably I'm not "normal" within the markets under test. At what point what you say to a business "you might make more money like this, but don't do it anyway because it's awful"?
Apologies if I'm reading more in to your statement than was intended. I've seen enough of your work to know that you get both design and business hence I think that statement interesting!
Your question exposes a really important fact about split-testing:
People are complex. Different stuff works for different people.
The yellow highlighter thing tends to make people like us wanna hurl… it triggers immediate distrust. Meanwhile, information marketers still find that it works.
You can't know who a person is when you split-test against them.
More importantly, you need a LOT of traffic, and even minor temporary changes in traffic composition can affect the quality of your data.
You will see that traffic from HN responds to & converts very differently than traffic from Lifehacker, and again different from Small Biz Trends. Initially you may find certain changes increase your conversion rates… and two months later you find you have lost almost all those customers, or that a disproportionate amount of them have asked for refunds, or been shitty to you in the support channels. Net result: a loss, which you only found out about months after you were so confident that your testing showed a gain. Would the other half of the split test do better? You can never know, because you cannot replicate the exact scenario ever again.
Traffic is not created equal; traffic is made up of different individuals. You can only ever get a feeling of what works for some of them some of the time, depending on several factors.
Take the yellow-highlighter example again: Why does it work for infomarketers when it makes people like us flee in the other direction? They sell to different people. Why would the Hacker News crowd be worse customers than the Lifehacker crowd? Because they are different types of people.
Unless you run a months-long test covering all parts of your business (refunds, avg turnover rate, length of customer stay, # of support inquiries/nastiness in support inquiries, resource usage, server bills, etc), tho, split-testing can't tell you which types of people are better for you.
And if you DO do that, when exactly will you have time left to expand and market your product?
I'm not saying split-testing is wrong or useless, but people use it to try to create scientific certainty when it's absolutely NOT possible to do so.
I'm not sure I'd agree with the last point about time - I think that balancing business generation, expansion and refinement is possible - but perhaps not so much amongst startup size businesses, which is more HN territory. In that case it should certainly be looked at as "is it better to try and get 0.1% more people to click buy or 20% more people to arrive" - or whatever you think your likely ratios are, hence cost/benefit call.
>Nobody wants to be told that they're wrong about their >most preciously held beliefs about the value of their skills.
That's one of the reasons I suggest testing everyone's design/interaction suggestions. Then I publish in internal emails the results of tests. It does wonders for reducing "preciously held beliefs". The number of bullshit, I-wanna-say-something-smart-too suggestions plummets and we're left with meaningful, committed suggestions.
>But for everyone who's championing split-testing, they >should remember a little bit about where it first became popular: Direct Mail.
Definitely. We studied this a bit in B-School and it was fascinating to realize that all that crappy, bizarre mail I get is often very carefully crafted. Catalog companies were doing split-testing on prices before vacuum tubes were invented. I hadn't considered this, but I should also be looking for demographic targeting cues in my mail just as I do on websites.
> A/B testing may only be as effective as the designs being tested
> As is true with any usability test, you gotta question
the motives behind the participants’ answers/reactions.
> A/B testing is typically used for tightly focused
comparisons of granular elements of an interface,
resulting in poor pastiches with results drawn from
different tests.
> Would you A/B test the Wii or Dyson or Prius or iPhone? Against what?
> A/B testing locks you into just two comparative
options, an exclusively binary (and thus limited)
way of thinking.
> A designer (and team) should have an opinion at the
table and be willing to defend it, not simply cave
into a simplistic math test for interfaces.
And binary thinking? Just because I do one A/B test doesn't mean the matter is settled. The "winner" of the first test can be tested against other options, whether it's right away or 6 months down the road. It goes on and on & hopefully we get closer and closer to "perfect".
What makes you say that? Did you split-test them? (Serious question.)
If someone holds an opinion but doesn't articulate the reasons for holding it, I'm willing to give them the benefit of the doubt. But if they try to articulate the reasons and the result is emotional and full of fallacies, I start to think that it's just rationalization and the actual reason is something else, perhaps some opaque emotional reaction that they're trying to justify instead of examining it.
That doesn't mean that what they are saying is incorrect, just that they are very bad at explaining it.
Ex: Most people "believe" in seasons, and evolution. But you ever try to get them to explain what makes those things WORK?
A/B testing is massively useful, but often misused in business. Commercially its become a bit of a buzzword. Which has led it being badly implemented.
I think designers often get frustrated when A/B testing is applied to their designs post-brief.
Designing a website for A/B testing purposes is a completely different brief. You are designing a toolkit for testing purposes. So that the next version will be informed.
A/B is quite is lot more complicated than some people appreciate. Ive often seen people create two completely different designs and then sort of let them fight it out. Yet they learn nothing from these experiments. There could of been elements that work well for certain persona's in each design!
Fundamentally, designer can be quite precious. They don't like criticism like this. After all its impossible to measure all the factors and how does one measure 'creativity'. Plus some will feel like this lowers there input. why not just throw all the elements in bucket and see which comes up trumps.
As a designer I love using this sort of testing. Design by data is a growing trend, that i think will get wider recognition. Sadly the A/B testing phase can create some slightly odd combination at first, but when you go back a re-design the page. You know its going to convert better.
That said.... A/B testing can be extremely dangerous:
- It's VERY EASY to design your test incorrectly. The most common result is that your test really ends up measuring some other factor of the design, or is chock-full of confounding factors. Any conclusions you draw from it will almost certainly be incorrect.
- It's EVEN EASIER to over-generalize from your data. People clicked on the green button more than the red button...people must like green buttons! Quick, redo the design with a green palette! These conclusions are always terribly tempting to make, since it feels like you've uncovered some deep truth about your users. You probably haven't, and it probably can't be generalized beyond the exact thing that you tested.
- It's easy to substitute A/B testing for critical thinking and self-editing. Two options? Don't worry about deciding! A/B test them! Another decision needs to be made? We'll let the users decide! The problem here is twofold: a). When designing something interactive, there are simply too many decisions to be made. You better get comfortable with making them, because there isn't enough time in the world to A/B test them all. b). (related to below) Your product will end up looking like it was designed-by-committee. Which it was.
- Related to the previous one, you can get trapped into doing "design by A/B testing". It's a simple algorithm. Get something that looks okay, then do an A/B test on some small changes. Pick the best one. Repeat. Unfortunately, the nature of A/B testing is that it is inherently isolated. If you're testing correctly, you've eliminated all your confounding factors. You're just looking at one aspect of your system. The problem is that interfaces are inherently tightly coupled. Everything tends to depend on everything else, and then you start throwing in abstract concepts like "feel" and "friction" and general "look" of the site (how well does it fit together? does it seem to have personality? Does it respect the metaphors it creates? Is it consistent?). A/B testing can't measure that kind of stuff. You also get into problems of local maxima. This is essentially a hill-climbing algorithm. Unless you're willing to occasionally start descending a hill, you're going to get stuck in an okay-but-could-be-better hump forever.
In short: A/B testing is powerful. Don't abuse it. And for godsakes, design your damn tests correctly.
In Ye Olden World of Direct Mail, everyone worked to beat the control. The control, your most successful mailing piece to date, was the benchmark against which everything was measured. You could modify the control, you could take a completely novel approach, but your work only counted for anything if it beat the control. Clients learned to ignore what they thought about a piece and care only about the results.
What the industry learned was that the only sane way to develop a brand new mailing piece was to imitate established controls. Books like Who's Mailing What and Major Mailers provided an index of successful mailings. Copywriters and designers came to understand the Darwinian nature of mailing and gave up on trying to predict what works. They learned that artistry was at best useless, at worst harmful. Their intuitions were completely wrong - the best-performing mailings were almost universally ugly, stupid and reliant on cheap tricks.
Outside of DM, designers almost never think in this manner. Throughout their education, they are taught to produce work that is aesthetically pleasing to the design world, with perhaps some concessions to accessibility. Aesthetics don't pay the bills.
The purpose of A/B testing is to measure the effects of a decision on the bottom line, but also to act as a driver for decisionmaking. A good DM designer has no hesitation in making a mailing piece look like a payslip or a bill, because he knows that such pieces work. His favourite font is Courier, his favourite colour is fluorescent yellow, his favourite adjectives are "splashy", "bold" and "eye-grabbing". He is blind to aesthetics and seeks only to drive the recipient to the next stage - to open the envelope, to read the headline, to the body, to the insert, to the coupon.
If you're looking to build a brand based on aesthetics, a designer is probably the right choice. If you just want to sell some product, 99% of them are a liability.
A vs B -> B
B vs C -> C
C vs D -> D
I don't buy the rest of the arguments.
HN is (at this time) programmer-centric. It ought to be obvious that you cannot design the best program on a large scale by optimizing functions. You have to start with a good design and then relentlessly optimize the bits. From time to time you have to consider larger redesigns that would throw all your previous optimizations right out.
Web/UX design is roughly analogous. Customer conversion, for example, is a minefield. You might have two archetypical personae, Alice and Bob. If you start with a design that appeals to the Bobs and the Alices equally, a small change (sexist image?) might alienate the Alices but increase the appeal to the Bobs. If the immediate gain outweighs the loss, it looks good.
But further gains run out of steam when you have all the Bobs but no Alices: You have reached a local maxima as you state so economically. Had you tried a different optimization that marginally increased appeal to both the Alices and the Bobs, you might be able to continue to acquire both the Alices and the Bobs over time, far outstripping the Bob-centric improvements that alienate Alice.
But you'll never know if you went with the biggest immediate gain.
He's wrong on one point, though: some designers definitely hate A/B testing because they don't like being proved wrong.
A fair number of designers apply a genius-beyond-fathoming approach to presenting designs. It works especially well for design firms; like many consulting companies they sell to people who can't really evaluate their output. As long as they act with confidence, clients think they're brilliant.
Those designers (whose inflexibility, lets be fair, we encouraged by putting them through waterfall hell) have a hard time adjusting to iterative processes. They have to stop pretending confidence in individual designs, and instead develop confidence in their ability to fail skillfully. They are forced to confront the fact that they aren't artists, but instead are professionals who are there to accomplish particular business goals. And they have to learn to compromise gracefully with people of different backgrounds.
Hating all that forced learning is perfectly natural.
First of all when you do A/B testing you don't ask the user. You test what composition get the most clicks.
Second A/B does not mean beautiful/ugly but for instance high contrast on call to action / low contrast on call to action.
Third you can't claim that there is no right design and then go on to claim that A/B testing is wrong because it forces you to choose between two alternatives. Then he is basically contradicting his entire claim.
Fourth. There are cases when design is important and cases where it's not. It's not always important and good design is not always beautiful in the way he seem to be talking about it.
At the end of the day better conversion must be the aim for any design. If you want to do art you should become an artist.
Design matters a lot, aesthetics less so. In fact aesthetics are a rather contextually relative thing and should be judged as such.
Only designers who think about A/B as the OP don't like A/B testing. There are plenty who understand that it can actually lead to better design.
In the end, aren't the users the only judge who matter? You hire expert designers to create an interface that your users judge to be pleasing, but also intuitive and easy to use. The user's judgement is king. If your users don't like your interface, it's game over. It doesn't matter how "cohesive" your designer thinks it is.
How do you A/B test novel interaction models, conceptual paradigms, visual styles (by the way, visuals & interactions have a two-way rapport, they inform each other, can’t separate them–see Mike Kruzeniski’s talks) which may vary wildly from before? Would you A/B test the Wii or Dyson or Prius or iPhone? Against what???
Rephrased: "I don't like A/B testing because you can't test everything using A/B testing." Huh?
A/B testing locks you into just two comparative options, an exclusively binary (and thus limited) way of thinking. What about C or D or Z or some other alternatives?
C'mon. The "A/B" in "A/B testing" isn't meant to limit choices. It's just a convenient way of expressing the idea. There's nothing stopping you from doing "A/B/C/D testing", or even starting out with 10 different ideas, and then using other methods to narrow them down to a few choices that you can then A/B test.
What if there are elements of A & B that could blend together to form another option? Avenues for generative design options are shut down by looking at only A and only B.
Sure, if you think of it in a ridiculously restrictive way. So do an A/B/C test between A, B, and some A+B blend that you think might work. Or make it an iterative process: do an A/B test, which let's say shows you B is better. Then do another A/B test, with the original B pitted against a B with some elements of A blended in.
If you look at A/B testing as the be-all-end-all and consider it the only tool in your arsenal, of course you're going to get bad results. But it has its uses in some places, and any designer who refuses to do A/B testing at all just seems uninformed and unwilling to examine hard data about how an interface might work.
I'm not sure any developers like any of those situations either.
"My data structure is killer!. What's that? Oh, we now need to track more info at a different level? But... that'll KILL all my ORM work! Users should just deal with what's there - it already works!"
The 'ugly pastiche' issue - I'm not sure why it'd be an ugly pastiche. The author seems to be suggesting that sections that 'won' from multiple disparate tests would be thrown together in to one interface without testing that interface. The little a/b testing I've seen done tended to be more iterative and sequential - design a vs b, then b vs c, then b vs d, then d vs e, etc.
Lately however I realized there is another metric which most A/B tests don't measure (but which perhaps could be measured). That metric is, if a person sees this variation of the page, how likely are they to tell people about the service.
For example if I include some sort of "human interest" content on my landing page such as my personal backstory on why I started the business, that might make it more likely to get mentioned in blogs. But such a backstory really doesn't mean much to people who are contemplating making a purchase and could hurt conversions. Not including it could deprive the landing page of thousands of leads. Including it could raise the conversion rate a bit, but there would be less potential customers to convert.
When a customer looks at the page, you are trying to make a sale. When a member of the media looks at the page, perhaps they would value different things, maybe in that case those site aesthetics would be more important.
BUT you should do A/B tests, if you do an A/B test with 1000k+ people and one design converts 20% more than the other, why on earth would you keep the other design? Just because the "professional" designer of the other design wouldn't feel bad?
Rather than question the underlying validity of the methodology, the author would be better served by questioning the validity of its implementation in the instances he's encountered.
Uh, you lost me at that point. Try an A/B test by varying "holistic cohesiveness or suggestive of a bold, vivid, nuanced vision that inspires users" with "consistent design integrity that inspires users".
Yes they are, actually. If your users don't like the design, the design is bad because the users are all that matter.
A/B testing may tell you to pick a "worse" design that results in better business. That's perfectly ok. There are plenty of design showcases one can browse for purely aesthetic appreciation of design.
I don't know much about A/B testing, nor have I done any. My first thought is: how do you avoid getting stuck on local maxima?
(The wikipedia article on A/B testing mentions A/B..Z testing and 'multivariant testing', which may be an answer to my question, but they don't seem to get much mention.)
http://method.ac/blog/design/devepers-designers-results.html
You only pick alternatives you'd be happy to see play out.