Hacker News Reader

Comment by jvans | Hacker News Reader

This is why I sometimes hate a/b testing. I'm sure someone at booking a/b tested these things and saw an increase in revenue. The thing that these tests don't measure are very long term effects where people either start to hate your product and look for alternatives, or become so numb to the changes that the initial novelty effect wears off. The person who ran the test gets a promotion for increasing revenue during the quarter but the net result is a massive negative for the longevity of the product.

View on HN

I have a personal experience where booking.com’s nudging caused me to reconsider my trip. I was trying to find something suitable to stay in Paris. Maybe it was the exaggeration of booking.com or maybe there was some truth, but at some point I shut down and made a 180 on my plans. I had realised that I don’t want to go somewhere where I have to compete against this avalanche of other visitors who were or were not snapping my accommodation options away. I am now visiting friends in another European city.

But did you still use booking.com for that other European city trip, or another platform?

I am staying with friends, who have a spare room available.

I had a similar reaction to Lyft’s “you have 2 minutes to accept this faster trip” prompt and emphasizing the faster, more expensive option first.

I saw that, balked a bit at the interaction and ended up taking a train instead. Not only was it $6.25 instead of $46 it got me there faster than Lyft’s fastest option. Including time walking to and from the station.

I wasn’t in a hurry but the in your face “look how much money people are willing to spend to save 5 min” helped me rethink my priorities.

Similarly (though also exhibiting teenage American language taking over the world symptoms) is Netflix's new (or new-ish, I just noticed it yesterday as a fairly frequent user) 'everone is watching' category. I have absolutely zero interest in that, it's literally repulsive, trivially incorrect, and just plain stupid even apart from the language - who cares what others have chosen to watch, how does it compare to what I've given thumbs up/down?

I join you in cringing at the word choice but I appreciate ways to find things that aren't similar to stuff I already like.

With a lot of recommendation systems, you watch and give the thumbs up to one piece of Danish art house cinema and forevermore that's all it will give you.

Plus it's more fun to watch something if there's a good chance you can find someone to talk about it.

Fair enough, but honestly in my experience Netflix is a long way off over-fitting to thumbs. I never watch cartoons or 'kids' category stuff, how hard can it be.

The 'everyone is watching' type reccomendations on youtube make me worry for the future of humanity.

Looking at the Youtube frontpage while logged out is like a sad, morbid, glimpse of the world.

Looking at the Youtube frontpage while logged in stares deep into your soul

(+ Youtube tries desperately to occasionally get you to also watch some open mouthed shock faced idiot in a thumbnail with $$$$$$$$$$$ in the title brought to you by the prank content creator house of the day).

I don't have an account, looking at the YouTube frontpage (which is presumably designed to appeal to some notional average user) is a seriously depressing picture of people/society.

I actually just got a message from Lyft this week when I was leaving the airport. I opened the app, checked the prices ($38!!), and closed it to catch the train instead ($7).

I quickly received a push notification saying "Save 50 minutes by taking Lyft. If you take transit, you will arrive at 12:36am".

Well, it was already quite late and it would be nice to save 50 minutes. So I booked the Lyft, and proceeded to wait for 40 minutes as over a dozen drivers would be assigned to me, see where I wanted to go (6km away), and cancel my ride. I gave up and cancelled the Lyft and only barely made the last bus. Lyft made my life a whole lot more stressful for no reason.

I was amazed to learn that 218 million people arrived in France during 2019 (tourist numbers indicate 90 million - not sure how the two numbers correlate).

Sources:

https://www.statista.com/statistics/436536/total-number-of-i...

https://www.diplomatie.gouv.fr/en/french-foreign-policy/tour...

That is not a problem with A/B testing. It's a problem with the values of the company. I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]."

People who value revenue metrics over all else will still do shitty things for users even if they don't A/B test.

The problem with that approach is you now have evidence that the short term change will show immediate results and no evidence of the long term concerns. Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.

> Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.

If this is true, that indicates that a company's theoretical "good values" are not being passed down to those decision-makers in a way that makes them impact decision-making.

Which means the company does not have those "good values" in the first place. They have lip service.

Values that are not practiced are not values. You are what you do, both when someone is watching and when someone is not.

I think I see where you’re coming from, but how I interpreted it, it’s just really hard to make these decisions, and from a managers perspective its not that black and white:

- almost any change will have have people arguing for and against it

- if ‘company values’ is a trump card to prevent a change, it will be used by the people against the change

- as a manager, to still make decisions in such an environment, you’ll find yourself needing to weigh the upsides and downsides even if there are strong company values (it just puts a higher weight on certain concerns)

- as parent said, short term impact supported with numbers is easier to weigh and defend than (possible but unknown) long term detrimental impact.

Thus, I think parent is right. Even in corporates with strong company values, it’s easier to prioritize the short term proven impact over long term unproven impact. And therefore, at scale, such decisions will be made.

Still, that’s what sets a good manager from an average, no?

No different with how there are relatively few good engineers but most are average or bad?

I mean there are companies that put out good products but most are average at best.

I suppose A/B testing is bad in the way that table saws are bad — that they are dangerous and I see a lot of videos of people using them in ways that are gonna chop off their fingers, but I wouldn’t blame the table saw for it or want to take it away.

You're assuming bad intent on numbers and good intent on gut. I've seen many places that just go with gut on everything, that's worse. I like working places that get numbers on everything.

You’re making a lot of assumptions. Looking at data and reacting to it absolutely does not imply a lack of values. What if the “good values” are to look at the numbers? This would be the right thing to do for the livelihood of employees, and the right thing to do from a scientific perspective. Some “good values” include never passing dogma down over time but instead questioning assumptions and testing for results.

This is where an involved founder can make all the difference. They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.

> They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.

And how do you know the short term changes will degrade the brand and the user experience?

Maybe deep down, users love dark patterns! Our judgement is powerless to determine; only the objective data can tell us!

In my experience founders are often more than happy to toe the line between nudges and shoves.

This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.

You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.

isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?

I suppose that is a way of looking at it.

Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.

Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.

And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.

That's correct.

Long term holdbacks actually are way way harder to implement and maintain in a fast moving product than meets the eye. We had passionate and ultimately unfruitful conversations about how to do that when I was there in 2016.

The only thing we had at the time was continued tracking against the experiment groups after flipping that switch.

For very long term effects (e.g., the impact of painting every surface of your website in ads which will certainly drive users away), hold backs aren't enough because it's too hard to consistently identify users. If ads will drive me away over the course of a year, there's a five percent holdback, and I use multiple devices and cash maybe be consistently identified for a few days or weeks at a time, that means that 95% of the time I use the website I get the bad experience, and I eventually bounce. The holdback will show no effect.

"Seeing like a State" in action.

Solid numbers of what? Here we're talking about numbers on short-term effects only, with zero data on long-term effects.

I'd claim that people who are reeds in the wind when there's a short-term gain available do not in fact have good values. "Take the money and run" is a value for sure, and it's common enough. But I'd have a hard time calling it good.

There is also a mathematical problem of assigning events and actions to long term effects. The usual IT crowd unfamiliar with the respective literature will try to ab test and grid search out of the actual scientific part of data science. I had also fallen to that trap.

What would you recommend reading on this?

There is a grey line. I think only one room left is actually useful info. But yeah they push it too far. But most people in a company will be able to argue for themselves the info is useful and truthful so in their minds it's morally ok.

Only one room left would be useful information if it were accurate. Most of the time, booking agents don't really have an accurate picture of inventory though. It's more like only one room available for booking.com to book right now.

It could be that the hotel is holding back rooms for other channels or because they like to not be fully booked so far ahead of time or perhaps the hotel has found listing only one room at a time gets them a better look to book ratio (in part because of anxiety inspiring features like this).

Without an understanding of the industry though, it's not really useful information.

DHH, founder of Ruby on Rails and Basecamp, drew a line between core values and A/B testing https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5...

> This is the tyranny of easy metrics. It's easy to measure how much money is saved by preventing cancelations, it's much harder to measure how much long-term business is lost by poisoning your reputation with the 99.9% of customers who had to jump hoops and dodge sleazeballs to get out of the subscription. But the latter could well be orders of magnitude money more over the long run.

I agree but a truly nuanced approach to interpretation of A/B tests is rare especially when mixed incentives are involved. Ignoring empirical evidence is bad and taking it as gospel is also bad.

Yeah-- I think blaming dark patterns on A/B testing is like blaming thermostats for chilly houses. A/B tests just show if one thing does more of something you're testing for than something else. If you're just testing for "conversions" then you're going to make websites like booking.com. If you're testing to see if users are more stressed out by one situation or another, or testing to see if expert users are stymied by some interface abstraction designed to make things easier for less sophisticated users, then that's totally different.

Sure, but it’s a lot harder to test for things like stress or being stymied than it is to test for conversion. Easy implementation + results that make money graphs go up will win every time without C-level involvement.

It's a lot easier to break something with a hammer than bang in some nails. Using a marker to deface property is a lot easier than using it to draw a picture.

>I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]."

Then why were they testing it, if they already knew other concerns would veto that alternative?

I suspect if you were to think a bit, you could come up with some ideas yourself. If you're really stuck, let me know what you've got so far and I'll fill in the rest.

I think I'll just wait for you to give the substantive reply you have in mind.

Accounts being jerks get less leeway from me than people being respectful and/or standing behind their words.

You'll be waiting quite awhile. If there's anything HN has taught me, it's the meaning of "pearls before swine". And the futility in providing free labor for people who seem to think they're entitled to it.

It's a discussion forum, amigo. You're supposed to share your insights with those who haven't yet achieved them. If you're not comfortable doing that, you can just not post instead of posting merely to whine and insult.

Bookmarking for later.

I share with people who seem like they'll benefit from it. That includes people who have done at least a little work to understanding what's going on. It definitely doesn't include HN's legion of querulous jerks who mainly look for nitpicks, gotchas, and other opportunities to feel smart, even when that comes at the expense of the good discussion this place is nominally for. You get to influence which group people see you as part of.

When I worked at eHarmony there were two things that we were aware of above all else:

1. Users hated all the advertising even if they were signed in with a paid account

and

2. eHarmony had no intention of stopping this because they valued the revenue from the advertising over the user experience.

I noticed that Redfin has started adding ads to their site which really annoys me. Folks you’ll get a sizable commission for connecting me with the listing agent. The ad revenue has to be a rounding error compared to that. Why are you damaging your brand by showing me ads for luxury vacations next to houses that I’m looking at buying?

Great examples.

To this list, I'd add Amazon's ads. It's providing them with very obvious revenue growth. It's parasitic on their ecommerce business, though, in ways that for me and many others, harm the things that made Amazon popular in the first place.

That'll never change, though, because that revenue has enough people attached to it that they'll fight tooth and nail to keep it no matter how much it harms the rest of Amazon.

Companies need to balance between metrics and vision. It's easy to chase metrics and lose track of why your company exists. If you're just chasing metrics, what differentiates you from anyone else? What big bets are you making? Every company I've worked at has fallen into this trap, and it's gotten worse due to the tech slump.

I think the problem underneath is quantitative fallacy: https://en.m.wikipedia.org/wiki/McNamara_fallacy

We had someone from Booking.com come and speak at a company I was working at a good few years ago to talk about their testing process. They were using the multi-armed bandit approach[1] of just throwing dozens of changes at the wall and seeing which worked best. It definitely reflected in the UX.

1: https://en.m.wikipedia.org/wiki/Multi-armed_bandit

“Dozens of changes” but without values who gets to decide WHAT changes are tested? Sure there is UX/CRO research but without a moral compass it is exactly how you end up with dark patterns.

Agreed, and my impression was there wasn't a lot of scrutiny there. It was purely numbers driven, if it could be tested and measured it would, and if it won it was adopted. Very little emphasis on UX, product cohesion, or any specific design principles.

Can’t believe they are giving talks on this like it’s innovative or something when admiring. I’d honestly be embarrassed saying that my design process was to make shit up and see what happens instead of investing in UX design/ research professionals can run generative studies to better understand their consumer base and sentiments towards booking trips online, like Airbnb heavily invested and can be seen as having a world class user experience next to booking.com

I digress…

It is also one of these things where the first to do it gets the advantage.

If you are the first to do the "only 2 rooms left" trick for example, you will get the full results before people get desensitized. But people will get desensitized everywhere, not just on your website, so if a competitor tries to pull the same trick, he won't get the same effect as you did. If fact, it may be time for you to roll back, to make competitors look bad for using the now well known and ineffective trick you invented. And if it works long term, then you get a head start.

Yup, this "poisoning the well" effect is real, and it's blatantly uncompetitive - which is why it's so important for regulatory agencies to step in and act fast and hard so that this "first mover advantage" is eliminated.

The problem is, regulatory agencies are slow as molasses and courts are overloaded with crap, which means by the time the process is done years later, the companies have long since switched to yet another sleazebag tactic.

Why? The tactics burn themselves out.

I don't want government agencies involved in dictating the minutia of acceptable UI design for a private website.

Businesses have brand reputations. Just let people vote with their wallets. That won't discourage all dark patterns, but is preferable to a world of over-regulation.

Businesses, particularly in oligopoly/platform scenarios, don't care about brand reputation. Just look at Meta - they don't even care about billions of euros worth of GDPR fines, it's all priced in.

There's no point letting "people vote with their wallets" when companies have monopolies or near-monopolies due to network effects - and these shitty tactics work, so their most effective competitors are practically forced to use them to compete.

See also: Google, Amazon, etc. The laissez-faire voting with wallets approach has failed time and again. It doesn't seem to discourage dark patterns at all. I'd love to use companies with less infuriating interfaces but they can't compete.

I'm not too concerned about over-regulation. I don't think there's much innovation possible in the renting rooms to the public space; the biggest change I've seen recently is in more dark manipulation of customer behaviour. However, perhaps there's an alternative to having governments regulate UI design, like requiring 3rd-party API access.

Booking.com lost me as a customer for life after I fell victim to their sleazy tactics a few times. I have refused to book anything on booking for the last few years because I didn't want to be mislead into booking a crappy hotel by their algorithms again.

I guess my decision doesn't matter much in the grand scheme of things, because they are still around and judging by the screenshots it's as bad as ever, and people still use the site...

Yup. Go now to booking.com and pick a remote roadside motel in Wyoming for a midweek stay sometime in March of 2024.

And watch booking.com try to tell you that "19 people have booked this today" or some such bullshit.

I’m surprised California hasn’t outlawed this tactic as deceptive marketing or such? Any idea why companies can falsify data for the sole purpose of deceiving consumers to increase their revenue?

There are no laws against it in the US. In the EU there are some[0][1] now.

[0]: https://www.acm.nl/en/publications/bookingcom-commits-adjust...

[1]: https://www.acm.nl/en/publications/booking-and-expedia-infor...

I've just tried this and now they show me identical room for the same price with two different nags. Not sure if they have only 2 or 5 (or 7?) "Queen Studios" left.

https://imgur.com/a/oqOTYlz

> watch booking.com try to tell you that "19 people have booked this today" or some such bullshit

...while watching uBlock Origin go crazy blocking stuff.

I really startet to hate using Booking.com, especially because every time after using it, they start bombarding me with emails. I could probably turn them off.

Another thing that makes me laugh now: I often go to the same hotel, and Booking.com provides better rates then booking directly (no idea why, I asked multiple times for the same discounts). And for my favorite category the hotel has only one room. So booking.com constantly warns me "only 1 room left!". Yeah, I know, there is only one ;)

> (no idea why, I asked multiple times for the same discounts)

Hotels sell a fraction of their inventory cheaply to resellers.

Also, they oversell, because people cancel. And when the hotel is oversold, and has to cancel a reservation, guess which customers get their booking declined?

When you are on booking.com you have the most choices. Not only between all the inventory they have, but you can also browse Airbnb.

When you're at the hotel and you ask their rate, your basically committed. They know how much time and money it cost you to drive to a different hotel. So they can set their rates accordingly.

And hotels hate reservations made over the phone or by email because too many guests never pay a deposit and never show up.

"And hotels hate reservations made over the phone or by email because too many guests never pay a deposit and never show up."

I'm thinking you're from a non-western country, otherwise that would be false; the only wiggle room you gave yourself was 'booking through email' which I don't think even really exists, unless you meant booking through a hotel's website. Hotels take your deposit when you make a reservation over the phone or on their website, and if you book through booking vs the hotel the hotel will lose 15-30% of the room sale.

You have just as many choices when looking for hotels, just look at google or bing. You don't need booking for anything. People rarely just drive up to hotels without checking hotels in the area beforehand on their phone. I'm not familiar with it in developing countries.

Even in Europe there are many smaller boutique hotels who cannot charge your credit card based on a phone conversation. Their bank either require them to use a speed point machine with the card present. Or use an online payment gateway with 3D secure it similar technology.

So they require the deposit as a bank transfer. And this is where the guest gets second thoughts.

People do actually go on road trips. "Oh look how nice that hotel looks. Let's stop and see if they have space for us". They get quoted a high price and that's when the hotel make profit.

I'm actually finding that to no longer be the case... just booked a hotel for a holiday and booking directly through the hotel's website was around 15% cheaper (and didn't try to charge me 50% extra for an infant). The hotel's payment flow for Amex was broken, but well that's another issue ;)

In theory, the solution in situations like this is to continue the test over a longer time. Typically with ‘holdbacks’ – subsets of users who don’t get a feature for a long time. This is easier if you have an app that everyone uses because with a website, it’s harder to reliably find holdbacks who are also a representative sample (eg it won’t work so well to hold back everyone using the site in a certain language as those people may be statistically different from the general population in other ways).

There are still a bunch of problems – higher maintenance burden, harder to iterate on a site quickly. Though I think you identify what I would consider the bigger problem which is that they cause political difficulties as a holdback can only really turn around and say that the positive impact people claimed wasn’t really borne out in the long term. So even at places that do holdbacks, the results may be silenced or ignored. If a holdback shows something continuing to work, that’s hard to get excitement about even though I think one should expect many of these a/b test results to not have long lasting effects.

The types of effects you want to measure would take months or years to show and are the combination of many different small decisions. Teams need to apply common sense thinking, empirical data, and the willingness to wrestle with uncertainty. All of that is hard, blindly following a/b test results relieves people of that cognitive burden

Am I the only one who thinks THAT is a dark pattern? It would at a minimum, confuse me, if I had a different UI / feature set than my friend(s) - confused for being unaware why my experience looks different, and you better bet that 90%+ of users do not know what an A/B test is. At worst, I'm angry because they are getting a better experience and I randomly got shafted with no ability to upgrade.

That's the same kind of thinking that would criticize pharmaceutical companies for giving placebos. They do it because they don't know with confidence, and they want to. They are collecting evidence.

To get into a medical trial you a) know it's a trial b) get informed that you might get a placebo c) consent to participate in the trial.

If a pharmaceutical company was discovered filling 30% of pill bottles they sent to pharmacies with placebos, they'd be sued unto the eighth generation.

Why is a long-term holdback ethically questionable in a way standard a/b testing is not?

Maybe that’s undesirable. I don’t think it’s a dark pattern as it isn’t about manipulating/tricking people into doing what you want.

Many companies in my experience do test for these things. They remove old features and measure impact periodically, or run a long term hold out bucket, or some other such approach. The deep dark secret of the web is that there often there isn't a negative impact that can be measured no matter how much people try to measure one.

One of the effects this has on me is that I will use booking.com to find places (amongst other tools), but book directly with the accommodation. Only if the accommodation doesn't do its own booking will I use booking.com to book (about once every twenty places).

I also had great results just walking up to the reception desk and booking there. Some deals are too good to be put online.

But when you are taking vacations with your family you don't want the stress and the risk of not finding a good accommodation.

Every time I've checked this, direct booking was more expensive than Booking.com or Hotels.com.

It often is, but I still book direct because I want to be a customer of the hotel and not of some third party. When things go wrong, I don't want to be be told to take it up with the third party (and this has happened to me).

I book directly. If there is a problem with the reservation, hotels have much more flexibility to accommodate changes. You can usually cancel on the arrival date with no penalty, change room type, etc.

I've heard others say that if you quote an online price, the front desk can match it (better to get 100% of the discounted price than 60%), although I haven't tried it myself.

I find this kind of feels like a squeeze on one's attention.

Where every app or signup has this false premise seeded in a design that it alone is at the center of your digital existence and therefore you can do anything it takes to vie for and keep attention and engagement because the product may experience too much attrition otherwise.

1) Take color away from sites who abuse it and watch your attention and focus go through the roof.

Grayscale the Web is a handy extension. https://chrome.google.com/webstore/detail/grayscale-the-web-...

It has a setting to just turn off all color, or just a site.

2) Add a firewall layer when signing up. I recently registered a voip number and an extra email account to check out for things.

It stops explosions of emails and text messages on my space of the same try to chase people into engaging in a pipeline.

It's surprising how few of these well oiled machines can do without being able to interrupt you constantly.

3) Notifications off, and checking them scheduled.

A good way to test any product beyond it is to silence all notifications. Maybe I have a habit to keep going back, or not.

Spot on! I think we have a similar problem with TV / video ads. Many of them are funny or intriguing the first time you see them (so they perform well in "one-shot" tests and focus groups), but after five views they start getting exceedingly annoying. A counter example is a State Farm's ad with a guy sitting quietly on a couch and watching TV for a few seconds; I find it very effective.

> I'm sure someone at booking a/b tested these things and saw an increase in revenue.

I know people who used to work at Booking.com. That's exactly what happened. From what I can gather, you're not allowed to make changes to the site without running a test.

While I agree, the problem is in the data-driven decision-making.

There is simply no good way to get good data for this long term type of effect. It's either going to be user interviews, some arbitrary score like customer satisfaction, or maybe a really convoluted split test (long ago cohorts vs new ones... but let's try to remove criteria so it's not apples and oranges).

If a company culture only values data in its product decisions, this is always going to happen.

Good luck fighting an argument against someone who has a supporting data point, and a revenue increase!

Real-life stores do this as well - make individually profitable changes (local maximums) that eventually make the whole experience so miserable that you no-longer want to shop there. Direct upselling is the most obvious, along with constant messing about with pricing and offers, as well as constantly pushing lower quality expensive own-brand goods, but the one I notice most is various cynically high pricing of non-core impulse items around checkouts - it tends to stick in my mind as the shop being expensive.

I don't disagree with you, certainly some "improvements" only have short-term positive effects. However, at least the A/B test was performed and data collected. Your assumption that it will eventually be "a massive negative" is pure speculation with no data behind it.

The value of a "brand" is really, really difficult to measure objectively, but certainly has an effect on your revenue. It's hard to know in advance what's going to tarnish your reputation in the long run, but by the time you can measure it, it's far too late.

Part of the reason you should always have a long term holdout, so you can see how the win degrades over time, which with dark patterns, it can do.