Hacker News Reader

Comment by causi | Hacker News Reader

The problem with AB testing is that it's a short-term strategy. For example, if a news site runs AB testing with headlines, they'll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles get the most alluring headlines.

View on HN

This highlights the major downside to "data-driven" policy and decisions.

Data can "lie". What is observed is not always reality, simply what we can see of it.

Consider auctions. You never actually "see" the bidder's demand or utility. Yes, there are some ways to structure auctions that in theory show willingness to pay and such (ignoring confounding factors and irrationality), but you don't actually observe anything beyond the bid.

Similarly, on websites, you don't always know the causal reasons people click here or there. You know perhaps enough to predict a step-wise behavior, but don't (usually) understand the full behavioral lifecycle -- especially if a metric improves but at the hidden cost of decrements to conversion and similar.

There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made the statistics more complicated.

Sure, but in many cases, such as the example given by GP, long-term AB testing is hard or almost impossible. For the testing to have validity, you need the A and B cohorts to be stable, and have little or no overlap, and that is hard for long time spans for anything that is not account based (and somewhat dangerous even for account-based things, as people will almost certainly start to notice that they are getting a different experience than their peers, which may upset them).

In online dating, at least, this is a non-issue. Using an online dating app is, ironically, a solitary enough activity that people don’t go around comparing whether their UI is different from their friends’ UI. You of course can’t let the same user see two versions, but that just means doing permanent group assignment on signup. We used to A/B test subscription prices over enormous ranges (e.g., randomly giving some people 90% discounts) and approximately nobody noticed outside of obscure Reddit threads.

I wonder if you two are talking past each other a little. I'm thinking that A/B testing for content is a different beast than A/B testing for experience.

I’m not disagreeing — My point is really, “not all AB testing is bad, even if the kind you’re most familiar with leads to shitty content.” My second comment was just more of side note.

Sure, this is a niche with some very nice properties for this type of thing.

OkCupid has completely destroyed its interface and utility, so whatever they’re doing doesn’t seem to be working anymore.

I left in 2015, as soon as it became apparent the party was over. OkCupid went downhill for a lot of reasons, but overly aggressive A/B testing wasn't one of them.

What do you think the reasons were?

Did you A/B test the matching algorithms?

Don’t recall specifically. There maybe may have been randomized A/B tests on the “special blend” at some point, I think — it was never spelled out on the site, but I think that was the experimental mix du jour and we tended to use that instead of forced randomization.

I write A/B tests for headlines for a news site, this is too broad a generalization. Clickbait titles aren't great for building subscribers or establishing trust, which is what we really care about (LTV). To the author's credit, our deepest testing insights come from analyzing a lot of historical data (not just last week's).

I’m a huge fan of metrics. Huge! But they are worthless when not combined with qualitative experience. AB testing needs to be combined with human-centered “actually talking to people about their experiences.” Otherwise, you drift and the metrics no longer match the objective.

> but the constant use of clickbait headlines will over time destroy overall traffic to your site

I'd add a bit of nuance here. They are very good at driving traffic, but very bad at building an audience. You do this long enough and your news site is now optimized for attracting hot-take appreciators who engage with the news like a tabloid. This drives away everyone who doesn't want to be reading a tabloid and makes you more dependent on keeping up with traffic-gaming strategies to continuously drive traffic. You've basically shifted your business from being a place that produces journalism to being a place that figures out ways to game social media trends and SEO.

Indeed,

If do more ad placements increase revenue is the test and then there is 20% jump what are you as an engineer going to do? Tell to management that its bad?

Reply on news.ycombinator.com