Hacker News Reader

Comment by lylejantzi3rd | Hacker News Reader

The problem with that approach is you now have evidence that the short term change will show immediate results and no evidence of the long term concerns. Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.

View on HN

> Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.

If this is true, that indicates that a company's theoretical "good values" are not being passed down to those decision-makers in a way that makes them impact decision-making.

Which means the company does not have those "good values" in the first place. They have lip service.

Values that are not practiced are not values. You are what you do, both when someone is watching and when someone is not.

I think I see where you’re coming from, but how I interpreted it, it’s just really hard to make these decisions, and from a managers perspective its not that black and white:

- almost any change will have have people arguing for and against it

- if ‘company values’ is a trump card to prevent a change, it will be used by the people against the change

- as a manager, to still make decisions in such an environment, you’ll find yourself needing to weigh the upsides and downsides even if there are strong company values (it just puts a higher weight on certain concerns)

- as parent said, short term impact supported with numbers is easier to weigh and defend than (possible but unknown) long term detrimental impact.

Thus, I think parent is right. Even in corporates with strong company values, it’s easier to prioritize the short term proven impact over long term unproven impact. And therefore, at scale, such decisions will be made.

Still, that’s what sets a good manager from an average, no?

No different with how there are relatively few good engineers but most are average or bad?

I mean there are companies that put out good products but most are average at best.

I suppose A/B testing is bad in the way that table saws are bad — that they are dangerous and I see a lot of videos of people using them in ways that are gonna chop off their fingers, but I wouldn’t blame the table saw for it or want to take it away.

You're assuming bad intent on numbers and good intent on gut. I've seen many places that just go with gut on everything, that's worse. I like working places that get numbers on everything.

You’re making a lot of assumptions. Looking at data and reacting to it absolutely does not imply a lack of values. What if the “good values” are to look at the numbers? This would be the right thing to do for the livelihood of employees, and the right thing to do from a scientific perspective. Some “good values” include never passing dogma down over time but instead questioning assumptions and testing for results.

This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.

Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.

That's correct.

Long term holdbacks actually are way way harder to implement and maintain in a fast moving product than meets the eye. We had passionate and ultimately unfruitful conversations about how to do that when I was there in 2016.

The only thing we had at the time was continued tracking against the experiment groups after flipping that switch.

And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.

For very long term effects (e.g., the impact of painting every surface of your website in ads which will certainly drive users away), hold backs aren't enough because it's too hard to consistently identify users. If ads will drive me away over the course of a year, there's a five percent holdback, and I use multiple devices and cash maybe be consistently identified for a few days or weeks at a time, that means that 95% of the time I use the website I get the bad experience, and I eventually bounce. The holdback will show no effect.

You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.

Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.

isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?

I suppose that is a way of looking at it.

This is where an involved founder can make all the difference. They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.

In my experience founders are often more than happy to toe the line between nudges and shoves.

> They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.

And how do you know the short term changes will degrade the brand and the user experience?

Maybe deep down, users love dark patterns! Our judgement is powerless to determine; only the objective data can tell us!

Solid numbers of what? Here we're talking about numbers on short-term effects only, with zero data on long-term effects.

I'd claim that people who are reeds in the wind when there's a short-term gain available do not in fact have good values. "Take the money and run" is a value for sure, and it's common enough. But I'd have a hard time calling it good.

"Seeing like a State" in action.

Reply on news.ycombinator.com