A/B testing gets misused to juice metrics in the short term

This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page, and some stupid interstitial pops up. Dude, I'm trying to enter my credit card information! Back off! Especially stupid for a "sign up for our newsletter" popup; we all know that unclicking the "yes, we can email you every 17 seconds from now until the heat death of the universe with valuable offers from 'our affiliates' which we define as 'anyone we share a species with'!" box on the checkout form is mandatory, and if we don't see it immediately we'd best go hunting for it. You've already default populating the checkbox to "yes" on this very screen, get out of the way!

Less unbelievably stupid, but related, is when I'm examining product X and just after I scroll down a bit to read more you pop up something related to... well... anything other than product X! I'm signalling interest in product X as hard as I can, and you've AB tested that this is a great time to jangle your keys over there instead? Your AB testing is stupid and can't possibly fail to be some stupid statistical fluke or other terrible error. What fisherman goes out on his boat, hooks a fish, and then rushes to throw another completely different lure out to the hooked fish and get them on that hook instead? This is another good touchstone for being "overoptimized".

A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a big part of that was the constant A/B/C/D/E/F testing. I never felt like I was using a complete product, everything felt like a constant beta that could be changed or rearranged at any point, and I was just doing free testing work for them while they harvest all my data.

Every app update was a risk of the app rearranging itself, or features appearing/disappearing. Eventually it didn't even come from app updates in the Play Store, and new interfaces would just appear one day when a server somewhere marked your account as being in the group that gets the new UI. This app that you were familiar with could at any point be rearranged when you open it on any given day. Then maybe a week later you open it and it's back to how it was before. A button you thought was here suddenly isn't, and you question whether something actually changed or if you're losing your mind. It's a subtle gaslighting that eventually I couldn't stand any more.

To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. That's not to say a company like Apple is creating products out of the goodness of their heart, but at least it feels like it was developed by humans who made an opinionated call as to what they thought was the right design decision, and what they would want to use. And in my 2 years of owning an iPhone, I've never opened my reminders app to find out that it's completely unrecognizable, or my messages app has been renamed or rethemed for the umpteenth time.

The main issue is that people mix conversion with customer obsession! Whenever you work on a product or feature you should be asking yourself "Is this really good for my customer" - if the answer is no, then no matter what the A/B tests/conversion rates show you don't do it.

Unfortunately we mostly hire the wrong people as PMs, who then hire clones of themselves. They are not truly customer obsessed and use A/B tests incorrectly which results in products that trick or force customers to do things they don't understand/want to do. Long term this is bad for the product and company

Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through.

Early stage startups should never A/B test. When you're searching for product market fit, you're doing global optimization within the search space. Your product will change drastically as you make new learnings. Premature optimization (A/B tests) will only be detrimental.

This is an interesting contrast to Amazon that also makes checkout easy but bombasts the user with thousands of listings, mostly mildly fraudulent and consisting of absolute crap, and still somehow gets repeat business.

My second biggest issue is: it's rare that companies offer actual, live-human support these days anyway. When marketing adds A/B testing, shit becomes really annoying if something breaks as a result - usually the phone lines are suddenly flooded, the agents have no idea what has happened either and try to reproduce and figure out what's going on (and sometimes can't because they aren't part of the test group!), and so even people who haven't been in the testing group are going to be very pissed off.

IMHO, A/B testing without explicitly notifying the customers in advance should be banned by law, and that ban be harshly enforced. Customers are not guinea pigs, and with the rise of elderly people on the Internet this becomes an actual public safety issue (as ever-changing stuff makes it easier for scammers!).

The three key outcomes I observed from the relentless A/B testing were UI antipatterns, team burnout, and a well-attended conference talk about "how we ran 105 A/B tests in a year, and what we learned".

Every single time I warn them about how the bill of goods they've been sold with A/B testing is almost completely unattainable, especially in the way that they want to go about it. They won't magically start getting more conversions by changing a button color. Even if they start getting more clicks, they rarely start getting more complete conversions, because the increased numbers is usually from people who weren't good leads in the first place.

On top of that every company I've worked with has no idea what the real methodology for good tests is, no matter how many times I explain it or put it in a slide deck. I would constantly get requests to use A/B testing for feature rollout.

Them: "Hey, could you do an A/B test of our existing site design and our upcoming redesign?"

Me: "if the old design performs better are you going to toss out the redesign?"

Them: "No we're going with the redesign but we want metrics on how it'll affect traffic"

Me: "Those metrics are useless if you aren't going to listen to them, and if the results come back and the old design performs better, you're not even going to put it in a presentation because it's counter to your planned actions. There's literally no point in running this test"

Them: "Run it anyway"

The problem is it's not subtle at all; there's a handful of those features that, when combined, end up being overbearing and noisy: "3 people looked at this listing within the past 3 days! 12 rooms left at this rate!" I don't care. I'm looking to book business or vacation travel. If a spot fills up I'll just go somewhere else. It'll be fine either way.

I don't use them anymore for that reason. Old soul (me) is old. (I'm probably in a minority, judging by their advertisement budget.)

However entering e.g. client's information take a lot of steps, you are constantly clicking "Next" throughout these beautiful wizards and pages. After some time everybody starts to feel that there must be a better way.

What is the solution?

Spreadsheet import! Where you can just do everything in this "complicated" UI of Microsoft Excel, with formulas, and hundred buttons at once on the screen. Fill in hundreds of rows of information and just import it to the "beautiful business system".

And the funny thing is, I agree with this article. Both the content and the heading of this hackernews article:

1. Notification/scare spam can have long term retention ramifications. The previous generation of experiment platforms made long term metrics literally impossible to read. But now companies can use holdout groups and long term metrics like retention to give more clarity.

2. Even if you can read long term metrics that include retention, the scare/notification spam could lead to less word of mouth growth. For travel, I am guessing that you will be swayed to drive WoM growth more by differentiated inventory, reliable service, and cost, so maybe it's just merely annoying but not a risk to the business.

3. Notice that Airbnb's UI is very, very different from Booking, Expedia. We made a conscious choice to always make sure Airbnb came across as a "sincerely helpful friend" as a booking platform. An AB experiment showing that metrics improve doesn't mean that you have to launch it. You can look at those results and say, "this metric lift isn't worth how ugly it's making my site", and that's a completely valid choice. (a choice we made often at Airbnb)

The author's idea is that this short term gain damages longer term metrics. That sounds logical and agreeable, but that doesn't make it true. Not in my experience anyway.

Probably the people complaining the most about annoying UI patterns weren't going to convert anyway. Whilst those coming with a specific conversion goal to your site will convert even if annoyed in the process.

Anyway, the true root cause goes all the way to the top. When you give a team a 20% sales increase target and "deliver by next quarter or be fired"...this is what you get. If the executive level dismisses a healthier, more sustainable long term growth model, then there's pretty much no way to stop this.

It's so hard to stop because it actually works. It works short term and evidence that it harms long time is typically lacking or it simply isn't true.

Netflix auto play? Is that you? You were a hateful idea, no one liked you, yet you stubbornly hung on for far too long

Having left FB years ago, I now watch people "navigate" their site/apps with disbelief.

No, that doesn't mean A/B testing is inherently short-sighted. It's entirely possible to measure long-term secondary effects of an A/B test. Just save a record of treatment groups, and remember to come back and compare long-term metrics like LTV down the road. We do this all the time at my startup, and of the dark patterns that we've tested, we rarely see a long-term negative impact on LTV that outweighs the positive conversion rate impact.

If you want to make a valid argument against dark patterns (which is basically what 90% of this thread is trying to do), it's unlikely to be grounded in efficacy. This is coming from a business owner who spends seven figures monthly on advertising, constantly split tests, and is heavily invested in only making decisions that are in the long-term interest of the business.

Instead try to improve the customer experience, make better products, improve customer service.

What once was ads everywhere, is now psychological gaming.

I hope someone comes up with a Google Extension, and maybe Apple with a new "Access Website" mode.

These messages are boring to be honest. Once you noticed them everywhere, game over for me. Time to move on.

One time offers, limited time offers, mailing list signups, up-sells, and cross sells are time tested ways to increase sales dating as far back as radio era telephone and catalog sales.

Steve Madden is a perfect example of this. They sell undifferentiated popular shoe styles less expensive than high fashion but more expensive than knockoffs. They have to hustle you to get you on their mailing list (for 10% off your order) in the hopes that you'll make another impulse purchase later when you get a text or email from them. If they weren't as aggressive you might never make another impulse purchase with them again as there are tons of brands selling nearly identical products.

Some companies are just horrible at hustling so they actually get in the way of you completing your purchase. In a competitive market this is a self correcting problem.

They spent more time seeing the items and..didn't like the pics and conversion went down. In the end we reverted to the crap gallery we had before, they don't click it anymore and conversion went back up again..

You have to click through at least 10 pages of additional offers (and many extra price things that are added by default!) before you get to the actual checkout page.

Site owners: please stop doing this. You’re turning the web into a cesspit. You’re part of the problem.

AB testing is and always has been fish oil for management. The only things it can actually prove, are more easily identifiable by common sense. So wherever it actually works, it was probably a waste of time / overkill for evidence.

- sincerely, a business analyst

A/B testing right now is done on cohort basis and tests are ran for weeks to couple of months. This means where lifetime span of a customer is beyond few weeks and months, it's really not possible to tell if global maximum was missed.

I.e. you increase the number of promotional emails the customers get per week. You do it for 3 weeks and see that customers who got those emails had higher conversion. But you didn't get to see that customers who kept getting those higher number of emails completely unsubscribed after 3 months of pain. But by this time all customers are on higher frequency group so it's hard to tell what would be driving the unsubscriptions.

I'm no expert but here are some solutions:

1. You should have really delayed long running control groups. Preferably going well beyond average duration your customer sticks around. These groups should get onto new things a year after. But even then it'd be not possible to take out WHAT feature is affecting them, because in 1 year main group would have accumulated lot of features. But still something...

2. You should really have lots of secondary KPIs that measure things that affect long term KPIs. Sure conversion is better, but is time spent reading newsletters increasing? Are buyers feeling good about their experience with the brand... some of these KPI are more qualitative and can't be just automated.

what else?

I currently work in a game publishing company, here are 2 anecdotes from it

1. We run an A/B for game performance but we keep changing the bids for our games, and thus get varied quality of users, A/B tests don't really help in such a case 2. Once by mistake we ran the same creative on FB for 2 different ads.. both ended up having totally different metrics

It's also worth noting that there's no way in hell they actually know that with any sort of precision. No GDS has proper up-to-date knowledge of bookings from all the various sources that hotel reservations actually go through (they overbook airline flights). What they're really saying is that the small inventory of rooms that are reserved for them to book exclusively are almost gone.

So if you do testing and it gives you some kind of result, the crucial step is trying to understand what it really means, is there something we can learn from it.

Unfortunately, this is also the hard part that requires actual effort and intelligence and is difficult to scale -- and so is frequently skipped.

I used to use that app all the time, then kids happened and spontaneous hotel reservation became rare. Fast forward a few years and a circumstance came up that made me think "Hotel Tonight". I discovered it wasn't installed on my new phone so I grabbed it. It was unrecognizable. Maybe the prices were as good, maybe it could still be used the way I used it previously, but it looked like it turned into a hotel booking app when what I wanted to see was a small selection of good hotels nearby with unusually low prices. One of the features was the lack of choice.

I finally opened the inspector and deleted it, so that I could use the menu to select "order online", which took me to a page ... with the same modal.

I agree with the sentiment on AB testing but I think the bigger insight is that we need to be reminded to see the forest for the trees with any process, tool, or goal.

Sometimes these intangibles are hard to measure and almost need to be sensed.

It reminds me of how you can see the exact same development methodology used at two different companies, where at one company it works beautifully and at the other it becomes a bureaucratic albatross.

For example, you can see if Group A or Group B from a test are more likely to still use the site 1 year later.

You hypothesize that those ways to 'juice the metrics in the short term' hurt the user experience in the long term... Well if your hypothesis is right, these long term AB results should show it.

No one EVER tests for mean-reversion over time.

17 years I've seen companies do A/B tests. I doubt I've seen a single convincing, durable result the whole time.

Saying you made a 10% purchase rate improvement in a month is an easy pay rise.

It happens when people change perspectives from building and sustaining businesses to exploiting and squeezing every employee, supplier, and customer for the last drop.

If you known its inside anatomy you know what I mean.