Hacker News Reader

Comment by wpietri | Hacker News Reader

That is not a problem with A/B testing. It's a problem with the values of the company. I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]."

People who value revenue metrics over all else will still do shitty things for users even if they don't A/B test.

View on HN

The problem with that approach is you now have evidence that the short term change will show immediate results and no evidence of the long term concerns. Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.

> Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.

If this is true, that indicates that a company's theoretical "good values" are not being passed down to those decision-makers in a way that makes them impact decision-making.

Which means the company does not have those "good values" in the first place. They have lip service.

Values that are not practiced are not values. You are what you do, both when someone is watching and when someone is not.

I think I see where you’re coming from, but how I interpreted it, it’s just really hard to make these decisions, and from a managers perspective its not that black and white:

- almost any change will have have people arguing for and against it

- if ‘company values’ is a trump card to prevent a change, it will be used by the people against the change

- as a manager, to still make decisions in such an environment, you’ll find yourself needing to weigh the upsides and downsides even if there are strong company values (it just puts a higher weight on certain concerns)

- as parent said, short term impact supported with numbers is easier to weigh and defend than (possible but unknown) long term detrimental impact.

Thus, I think parent is right. Even in corporates with strong company values, it’s easier to prioritize the short term proven impact over long term unproven impact. And therefore, at scale, such decisions will be made.

Still, that’s what sets a good manager from an average, no?

No different with how there are relatively few good engineers but most are average or bad?

I mean there are companies that put out good products but most are average at best.

I suppose A/B testing is bad in the way that table saws are bad — that they are dangerous and I see a lot of videos of people using them in ways that are gonna chop off their fingers, but I wouldn’t blame the table saw for it or want to take it away.

You're assuming bad intent on numbers and good intent on gut. I've seen many places that just go with gut on everything, that's worse. I like working places that get numbers on everything.

You’re making a lot of assumptions. Looking at data and reacting to it absolutely does not imply a lack of values. What if the “good values” are to look at the numbers? This would be the right thing to do for the livelihood of employees, and the right thing to do from a scientific perspective. Some “good values” include never passing dogma down over time but instead questioning assumptions and testing for results.

This is where an involved founder can make all the difference. They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.

> They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.

And how do you know the short term changes will degrade the brand and the user experience?

Maybe deep down, users love dark patterns! Our judgement is powerless to determine; only the objective data can tell us!

In my experience founders are often more than happy to toe the line between nudges and shoves.

This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.

You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.

isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?

I suppose that is a way of looking at it.

Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.

Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.

And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.

That's correct.

Long term holdbacks actually are way way harder to implement and maintain in a fast moving product than meets the eye. We had passionate and ultimately unfruitful conversations about how to do that when I was there in 2016.

The only thing we had at the time was continued tracking against the experiment groups after flipping that switch.

For very long term effects (e.g., the impact of painting every surface of your website in ads which will certainly drive users away), hold backs aren't enough because it's too hard to consistently identify users. If ads will drive me away over the course of a year, there's a five percent holdback, and I use multiple devices and cash maybe be consistently identified for a few days or weeks at a time, that means that 95% of the time I use the website I get the bad experience, and I eventually bounce. The holdback will show no effect.

"Seeing like a State" in action.

Solid numbers of what? Here we're talking about numbers on short-term effects only, with zero data on long-term effects.

I'd claim that people who are reeds in the wind when there's a short-term gain available do not in fact have good values. "Take the money and run" is a value for sure, and it's common enough. But I'd have a hard time calling it good.

When I worked at eHarmony there were two things that we were aware of above all else:

1. Users hated all the advertising even if they were signed in with a paid account

and

2. eHarmony had no intention of stopping this because they valued the revenue from the advertising over the user experience.

I noticed that Redfin has started adding ads to their site which really annoys me. Folks you’ll get a sizable commission for connecting me with the listing agent. The ad revenue has to be a rounding error compared to that. Why are you damaging your brand by showing me ads for luxury vacations next to houses that I’m looking at buying?

Great examples.

To this list, I'd add Amazon's ads. It's providing them with very obvious revenue growth. It's parasitic on their ecommerce business, though, in ways that for me and many others, harm the things that made Amazon popular in the first place.

That'll never change, though, because that revenue has enough people attached to it that they'll fight tooth and nail to keep it no matter how much it harms the rest of Amazon.

DHH, founder of Ruby on Rails and Basecamp, drew a line between core values and A/B testing https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5...

> This is the tyranny of easy metrics. It's easy to measure how much money is saved by preventing cancelations, it's much harder to measure how much long-term business is lost by poisoning your reputation with the 99.9% of customers who had to jump hoops and dodge sleazeballs to get out of the subscription. But the latter could well be orders of magnitude money more over the long run.

There is also a mathematical problem of assigning events and actions to long term effects. The usual IT crowd unfamiliar with the respective literature will try to ab test and grid search out of the actual scientific part of data science. I had also fallen to that trap.

What would you recommend reading on this?

I agree but a truly nuanced approach to interpretation of A/B tests is rare especially when mixed incentives are involved. Ignoring empirical evidence is bad and taking it as gospel is also bad.

Companies need to balance between metrics and vision. It's easy to chase metrics and lose track of why your company exists. If you're just chasing metrics, what differentiates you from anyone else? What big bets are you making? Every company I've worked at has fallen into this trap, and it's gotten worse due to the tech slump.

>I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]."

Then why were they testing it, if they already knew other concerns would veto that alternative?

I suspect if you were to think a bit, you could come up with some ideas yourself. If you're really stuck, let me know what you've got so far and I'll fill in the rest.

I think I'll just wait for you to give the substantive reply you have in mind.

Accounts being jerks get less leeway from me than people being respectful and/or standing behind their words.

You'll be waiting quite awhile. If there's anything HN has taught me, it's the meaning of "pearls before swine". And the futility in providing free labor for people who seem to think they're entitled to it.

It's a discussion forum, amigo. You're supposed to share your insights with those who haven't yet achieved them. If you're not comfortable doing that, you can just not post instead of posting merely to whine and insult.

Bookmarking for later.

I share with people who seem like they'll benefit from it. That includes people who have done at least a little work to understanding what's going on. It definitely doesn't include HN's legion of querulous jerks who mainly look for nitpicks, gotchas, and other opportunities to feel smart, even when that comes at the expense of the good discussion this place is nominally for. You get to influence which group people see you as part of.

I think the problem underneath is quantitative fallacy: https://en.m.wikipedia.org/wiki/McNamara_fallacy

Yeah-- I think blaming dark patterns on A/B testing is like blaming thermostats for chilly houses. A/B tests just show if one thing does more of something you're testing for than something else. If you're just testing for "conversions" then you're going to make websites like booking.com. If you're testing to see if users are more stressed out by one situation or another, or testing to see if expert users are stymied by some interface abstraction designed to make things easier for less sophisticated users, then that's totally different.

Sure, but it’s a lot harder to test for things like stress or being stymied than it is to test for conversion. Easy implementation + results that make money graphs go up will win every time without C-level involvement.

It's a lot easier to break something with a hammer than bang in some nails. Using a marker to deface property is a lot easier than using it to draw a picture.

There is a grey line. I think only one room left is actually useful info. But yeah they push it too far. But most people in a company will be able to argue for themselves the info is useful and truthful so in their minds it's morally ok.

Only one room left would be useful information if it were accurate. Most of the time, booking agents don't really have an accurate picture of inventory though. It's more like only one room available for booking.com to book right now.

It could be that the hotel is holding back rooms for other channels or because they like to not be fully booked so far ahead of time or perhaps the hotel has found listing only one room at a time gets them a better look to book ratio (in part because of anxiety inspiring features like this).

Without an understanding of the industry though, it's not really useful information.

Reply on news.ycombinator.com