Hacker News Reader

Comment by jacquesm | Hacker News Reader

Those searches maximize their profit, that it comes at the expense of your utility is your problem.

The same with google: google search is going down the drain but google search income is up.

This is exactly the kind of thing that tends to enable competitors in normal markets, but on the internet monopolies are propped up by all kinds of tricks and then you can make moves like this without expecting to be punished.

View on HN

It seems like A/B tests failing to capture long-term effects is a common trend among these companies.

Assuming you’d want to work data-driven, what would be a better approach to figure out what works and doesn’t? Or is it a conscious decision to capitalize on the short-term at the expense of the long-term?

I think there's an aspect of the recently-discussed right-hand pole instability: an A/B test continues just long enough to see an increase in a given metric, but basically never long enough to determine whether that increase results in a change in behaviour that results in a longer-term net-decrease in that metric.

e.g.: Putting emojis in your notification texts increases engagement… until it kills your engagement entirely.

You dervserve what you get when build superstar teams of l33tc0de bots

I agree with the sentiment, but if you keep performing small A/B tests (including emoji vs plain) this would be discovered in the future as a performance gain and reverted. Assuming no manager says "we already ran that test".

Not necessarily, because once the result has been generally has been rolled out, you've started altering the population your experiments are drawn from.

No amount of short-term testing will discover [*the cause of] a long-term decline, if the long term response of the system of customer/product/company is swamped by an opposing reaction in the short term.

Exactly, there will be no way to determine that why negative effects you measure were actually due to the emoji.

The only real way to deal with this is to run these types of A/B tests “forever”, which is not desired. Which is why I’m very much wondering, what is the appropriate way to deal with this?

I agree with your point. Just to add to the conversation.

The population is already changing long term for reasons outside of our control. If for instance, other companies are adding the emojis and people are tired of emojis that will affect the performance of our experiments.

It is by no means perfect, but at least it offers a way to get feedback (given enough users, not annoyed enough :)

Reply on news.ycombinator.com