Investigating how the New York Times A/B tests their headlines | Hacker News Reader

Investigating how the New York Times A/B tests their headlines

blog.tjcx.me

248 pointsmtlynch120 comments

This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time.

But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half will see an alternative headline, for about half an hour."

So given that the author mostly observes long consistent blocks of time with the same headline, that suggests the NYT is allocating them to a subgroup in a persistent way (by IP or whatever). Then perhaps the cases where they didn't observe A/B testing were just cases where they were randomized into the optimal (hence final) headline subgroup by accident at the beginning, and never saw any different.

+1, they didn't explain that at all.

In addition, there's only one case too where it changes back to the original and that's a minor In, in change.

Edit: The only example of clickbait here is OP's title.

Agree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of the Egg McMuffin, one from 7a-11a and the second from 11a-6p, and declaring the v1 a clear winner because it had more sales.

Assignments are also almost certainly sticky based on a browser cookie.

No. It’s like if McDonald’s offered a in the first half hour, and b in the second half hour.

I’m sure you would agree that would be a good representation.

The traffic to a restaurant isn't uniformly distributed over an hour window.

There's a peak 30-min in the morning just before work starts for most people, for example.

Sequential testing will never be truly random, unless you can guarantee that your traffic is uniform across time.

Imagine you're running an AB test on a website, where the number of customers you get increases linearly over time. If you test two features sequentially, there is no duration you could pick that is small enough to where you would not expect the second option to win.

Also, if the content of the story changes with the headline, then it is probably just reflecting a developing story. That's what the SpaceX and Cuomo stories seemed to be doing. Assuming that all headline changes are just because an algorithm thought they were more dramatic seems to be jumping the gun.

you’re right! as someone who has built (and builds the future of) the testing/algorithmic machinery at the NYT, this is one of the “misses” in the post.

Ok, I've consed 'investigating' onto the title above, which hopefully adds an appropriate degree of uncertainty. Thanks!

It took me a while to understand this, but I think the long consistent blocks are a sorting artifact. As the observed headlines are bucketed by the hour. You don't see the title flip within the hour, you just see the percentage.

It would have been more interesting to see the title flipping back and forth, because that would reveal how long those tests last. Less neat though.

OP here. I don't know if anyone reads old HN threads but thought I'd clarify some things:

1. My scraper runs every 5 minutes, with a randomly-generated user-agent and never sends cookie headers

2. The charts are bucketed by half-hour periods, so even if the headline flips back and forth many times in half an hour, the colors are grouped together

3. Agreed that in the SpaceX situation (and maybe the Cuomo situation) the headlines change because the stories change. But, e.g., in the Meghan Markle situation, the first headline appears _after_ the interview is over. But that's something to watch out for!

And charts like this one[0] look (to me) like a clear example of A/B testing. But would be interested to hear other explanations!

[0] https://nyt.tjcx.me/articles/1163e0c4-e609-5cfa-aff1-b0945f1...

Did you send from the same IP? (Worth including in the post either way).

They may also conduct certain A/B tests only on cookied users, in which case your cookieless bot would not have seen everything.

People read week-old posts on Fridays, when the hacker newsletter comes out. Thanks for the clarification!

Should a news source be optimizing for engagement, or for accuracy in communicating facts? I understand it's a business, but it's analogous to a bakery that labels all its goods as its best-selling items, only to confuse and disappoint the buyer when they open the box and find something different inside. It may sell that product that time, but it goes against the purpose of the organization as a whole, and doesn't seem like a sustainable practice.

This is my takeaway. There is a huge incentive for the media to make things more sensational or dramatic because it drives clicks and views.

I might also argue that online news drives this more so than print - with print you tend to buy the paper based on the totality of their reporting and reputation versus a zero-cost click of a headline on your screen.

Would you be more trusting of NYT (or other news sites) if they pledged not to look at engagement stats?

It would be a positive signal but it wouldn't be the only signal. You can still do untrustworthy things without engagement stats.

It’s fundamentally possible to have both. Although, yes, it’s very tempting to sacrifice one for the other, once you start to optimize.

Is it not the goal of a headline to engage?

Yes, but not at the cost of accuracy.

But multiple headlines can both be accurate with one being more engaging. These aren't mutually exclusive.

Article about a bank robbery:

"First National Bank Robbed"

"Gunmen Rob First National Bank in Daring Robbery"

My favorite this year was from the Times of India on the Jan 6th US capital insurrection.

"Coup Klux Klan: Don triggers mob & rob bid"

"Engaging" while being highly irritating. The first one is concise.

True, but which is more likely to get your click: a headline you're indifferent to, or a headline that fills you with righteous indignation? I think that if there's one thing that the development of social media has taught us about human nature, it's that any emotion is better than boredom, from an engagement standpoint.

That doesn't mean what OP has mentioned is wrong. NYT has a history of misleading headlines [1].

[1] https://thepostmillennial.com/nyt-makes-terrorism-story-into...

Is there actually an either/or here? Surely it’s possible to write two (or more) headlines for a story that are all equally accurate?

But, from the OP article, that doesn't particularly seem to describe at least some of these cases.

Yes, but did you read the examples in the article? Many of the high-engagement headlines ignored most of the content in favor of a tangential but titillating detail. That's National Enquirer stuff, not what the New York Times has prided itself on being.

I did, but honestly most of the examples confused me. Both the Tesla rocket example and Cuomo story are surely headline changes relating to the story changing? They didn’t add “Cuomo apologizes” to titillate, it’s because… he apologized, and he hadn’t when the story was first published. Which isn’t even A/B testing! The article also more or less states this, hence my confusion.

I think the same is true for the Trump CPAC speech article. The blog post compares the initial headlines about him “claiming leadership” to the ones about him attacking other Republicans. But just from eyeballing the chart, first the headline changes from “Plans to Claim” to “Claims”, presumably around when he started giving the speech, and then shortly afterward it’s changed to mention him attacking Republicans, probably at the point in the speech where he did that. There does seem to be an A/B test between the last two headlines, “Trump Pledges G.O.P. Unity, Then Attacks Republicans Who Backed Impeachment” vs. “Trump’s Republican Hit List at CPAC Is a Warning Shot to His Party”. But that’s not the comparison the blog post makes. And among those two, while the winning headline (the second one) is clickbaitier, both focus on the same part of the speech, and neither is inaccurate.

Are you saying there isn't just two sides to every story?

Yes, so A/B testing won't work. You need a one-armed bandit algorit.

Headlines have become advertisements. They're almost never written by the journalists who wrote the story, they're written for space constraints that change on a whim ...

They're very rarely "accurate" the way stories are intended to be for these reasons.

Have journalists ever controlled or written their own headlines? I don't think this is a change from any point in newspaper history.

Why has it been historically separated like this?

Because the headline isn't part of the story, it's part of the index/ToC of the whole paper, so it's written by the staff that put the whole paper together.

Which is to say, it's always been click bait.

The latency of decision cycle was too great in previous times. Even within digital workflows the length of time between knowledge of final space available on the physical page and when that space had to be filled with text was too short to haggle with a writer. It had to be performed by a smaller set of people right next to the people laying the type of not the people laying the type.

The story is worked on throughout the day/week. It goes through a number of copy editors and revisions. Its final publication date might be in limbo.

It's chosen for Wednesday in the budget meeting. It's 1A material. A cursory headline is written. Then a big news event happens: now it's 2A material. That means less space. The copy needs to be cut, the headline needs to be one line shorter, etc. This happens at, say, 9 pm and the journalist and editor in chief have been at home for hours.

Two reasons: the headline writers need to know exactly how many characters to use, as part of the prepress process, right in the layout room, while the journalists are out in the field, and 2) headline writing is its own skill (especially for tabloid headlines).

I wasn't suggesting that journalists wrote their own headlines, although for smaller publications it was more likely to be the case. I worked at a newspaper in a 50k city and journalists gave suggestions with their copy.

My point, rather, was that headlines slip past the journalistic barriers applied to copy. Often the people on the desk are making a headline fit at 11:45 pm or later and have no real oversight.

> Should a news source be optimizing for engagement, or for accuracy in communicating facts?

A journal is a business, with something to sell, the news, and the attention of their readers to advertisers as well.

> The New York Times is a big deal. As they tell their advertisers, the NYT is the #1 news source for young, rich thought leaders:

Obviously ultimately it reflects badly on the profession, since all these news sites are using the same clickbait techniques, from Breibart to the Dailymail to NYT, since they probably hire the same consultancies when it comes to clickbaiting design, or at the very least, people who come from the same marketing circles/education.

Do young people read the NYT?

A simple answer to a simple question: Yes.

> the same marketing circles/education.

Also "young, rich thought leaders", right?

It _usually_ makes sense to optimise for engagement, particularly in the US where there's no state-funded media outlet (cue discussion about socialism that will be ignored). The UK has the BBC, Australia has the ABC, and both are state funded media outlets that aren't (at least overtly) driven by views. I'm sure their funding largely depends on how many people are consuming their product, but it's not as though some editor is going to be fired that afternoon if their "Oprah/Meghan" story loses out to the Murdoch equivalent. I'd be curious to compare how A/B testing works for state-funded media vs pay-for-access media outlets

There’s absolutely state-funded media in the U.S. It just doesn’t encompass as much as those in the UK or Australia.

C-Span is a good example.

The Cable-Satellite Public Affairs Network[1] isn't publicly funded, no.

PBS, NPR, and, via the Corporation for Public Broadcasting[2], lots of individual, independent local radio and television stations.

[1]: https://en.wikipedia.org/wiki/C-SPAN [2]: https://en.wikipedia.org/wiki/Corporation_for_Public_Broadca...

npr gets like 2% of its funding from public sources, despite being national public radio. they're as beholden to moneyed and corporate interests as nytimes, and as partisan and outrage-oriented too. i stopped donating years ago.

I had to stop as well.

The article defending looting was absolutely a breaking point.

not sure which article was your breaking point, but for me, it's been a steady downhill slide for at least 5 years, with the complete whiff on the run-up to the 2016 election being a real wake-up-and-take-stock affair.

Based on the amount of click-bait from the BBC that gets posted to HN, I’m not sure your state funded option is much better.

I know the CBC in Canada is absolute crap - 80% of the reporting is sensationalistic and for the past 4 years has been focused on the nuances of Trump - not exactly relevant to Canada. At least not deserving of more coverage than domestic issues.

The BBC gets roughly a quarter of its income from commercial sources

https://www.icaew.com/insights/viewpoints-on-the-news/2021/j...

If people don’t like click bait I recommend not using news aggregators that only show the headline and are curated based on the votes of users who don’t read the articles!

You know of an aggregator that shows the actual headlines? The ones I read use a usually poorly editorialised headline created by the user.

I made zipnews.io, aggregates the news of the day on any topic and sends you a short blurb summarizing the top 5 stories. Headlines are included.

Sounds like someone needs to make an aggregator that shows only brief auto-generated summaries, no headlines or images.

Even better, these autogenerated summaries should give most weight to paragraphs at the end of the article.

I often find that New York Times and other media put important details that sometimes contradict the headline or challenge the main idea of the article somewhere close to the end of the article.

This lets them still claim objectivity while preserving a highly polarizing message.

This has a name its called burying the lead, its used because news papers are structured in a way where they know people have limited attention so usually they don't read to the end. All the important bits are typically laied out linearly, often trailing off at the end.

Burying the lead takes advance if this and puts the important but inconvenient information at the bottom.

FYI, the name of the introductory section of a journalistic article is "lede", not "lead", and the practice is called "burying the lede".

Yes and no. The GP probably knows the story that "lede" is a more recent spelling than you might think. e.g. This[0] article from Merriam Webster says "Although evidence dates the spelling to the 1970s, we didn't enter lede in our dictionaries until 2008." It says the different spelling was introduced to differentiate it from lead (the metal) which at that time was talked about often at newspapers re linotype printing.

That article links to this[1], by Howard Owens:

"It was then I realized, there is no historic basis for the spelling of a lead as “lede.” “Lede” is an invention of linotype romanticists, not something used in newsrooms of the linotype era."

He says he looked at books from the 1940s - 1980s, "The fact is, in none of the dozens of old journalism books that I have examined — none of them — spell it “lede.” I can’t find the definitive first reference to “lede” but it doesn’t start appearing in journalism books until the 1980s. .. It wasn’t until linotype was disappearing from newsrooms across the nation (late 1970s and into the 1980s), that we start seeing the spelling “lede.”

The safest conclusion, then, is that “lede” is a romantic fiction invented by those who were nostalgic for the passing of the linotype era."

[0] https://www.merriam-webster.com/words-at-play/bury-the-lede-...

[1] https://howardowens.com/lede-vs-lead/

Sure, lede is very recent, and largely without a solid historical basis, but it's the far more common spelling these days.

Journalist here. While I love that there's a discussion about burying the lede on HN, the practice you reference is known as a kicker. It's a last thought that is meant to twist, undercut, or add complexity to the narrative thrust of the story. The best ones add a ton of value to stories and reward people who read to the end.

I think user written, three sentence summaries for hacker news posts would be nice. I like that I can submit random interesting articles here, but sometimes a bit of context is necessary to get others engaged.

This is why I’m letting my NYT subscription expire. I see no difference between them and tabloids.

Sensationalist junk.

It's not an either/or. I work at a place that produces serious news. User engagement pays the bills and it also gets vital news in front of more eyes. Summing up a complex story into one sentence that convinces a reader to click and learn more is an art unto itself.

Except most readers don't go on to actually read much of the article. One of the problems I have with NYTimes (and others) is that the contents of an article, especially if you read the full story, are often times quite different than the assumption the headlines is implying. Afterall, who really has 20 minutes to spend long-form reading a single article? Better for the publication to just tell you what you need to think with a simple 10 syllable headline. (I'm joking here, but think it's a valid observation on others around me consuming the media).

The other problem I have, and why I stopped subscribing to them was their app was terrible and mostly only promoted opinion pieces to me. In fact, I'd love the OP of the story to study how many of the top 10 articles are opinion pieces at any piece of time.

It's the goal of Meredith Levien, their current CEO, for the NYT to be considered a "world class tech company"[1]. I believe doing A/B tests is a way to work towards that goal.

[1]: https://www.theinformation.com/articles/meredith-levien-want...

The article is what communicates facts. The headline is just an advertisement to draw the user in to read it. Optimizing the headline means the article is able to inform more people; in line with the mission of the organization. NYT wants to increase subscribers, not ad revenue, so it's incentivized to not confuse and disappoint readers with clickbait headlines.

If you're interested in observing this in realtime, there's a great Twitter account called @nyt_diff[0] that automatically posts when headlines on the front page change. Really cool to watch articles get updated as stories evolve. It's particularly interesting to witness headline writers grapple with how and how much to editorialize.

[0] https://twitter.com/nyt_diff

Cool bot. I'm a keyboard warrior and I've been burned by the NYT and other "mainstream" news site stealth changes multiple times. Now when a big story comes out I save the page using SingleFile to Dropbox and when the edited part comes up in the conversation I compare both versions copied paste with Copy PlainText and Meld which allows me to post nice screenshots of the diff.

Agh, this is using a horribly noisy diffing algorithm.

With quite a few of the posts I'd just present them side by side, with the differences highlighted.

There's a project inspired by nyt diff and NewsDiffs and working on improving the algorithm and supporting many more news outlets:

https://github.com/DocNow/diffengine

Valuable anecdata. News is defined by conflict, (literally, no conflict = not news) and this shows how headlines that express the most conflict get the most clicks.

The integrity issues come up when even though the facts reported are real, the conflict that frames them is manufactured - and this is why people reject news. They don't reject it because of fake facts, they reject it because of fake conflict. When I want the real news, I go to fringe websites, because they get the real conflict right, and if I need details and facts, I can look those up. The reason they get the real conflict right is because by definition the fringe lives in that conflict, and they are the real anti-establishment that creates a counter balance narrative, whereas a conflict produced by setting mundane events and facts against the backdrop of an ideology designed to manufacture conflict is unreadable tripe.

If I needed to sustain the dissonance of establishment narratives, I would read the NYTimes to keep up the appearances, but since my livelihood and aspirations do not depend on that, I have the freedom not to engage it. If you think you are being played and manipulated, watching these A/B tests should be enlightening.

Meh. This is the narrative of Marx and Foucault, and one I think is demonstrably false.

For multiple counter-examples, see pretty much everything on https://www.msn.com/en-us/news/good-news

Every single one of those headlines asserts a conflict, and the only difference is these MSN ones appeal to sentiment instead of outrage.

Arguably, newspapers as a concern are a post-Marx phenomenon, and it's what people get taught in j-school, and what editors accept. Students of both are peddlers of pernicious nonsense, but their students also produced some entertaining and powerful things, notably recycled recipes for accelerating and reverting democracies into their inevitable tyrannies.

Anyway, the NYTimes doesn't register as meaningful to me, and if they were looking for a reason why people are leaving them and other mainstream outlets behind, it's becuase their conflicts are contrived.

> literally, no conflict = not news

Read local news.

In fact that gives me an idea, a local news aggregator which ignores redundant news and can find the most impactful ones.

it's not simply conflict, but value-laden judgement and manipulative framings to try to coerce, not invite, agreement.

I’m really happy that folks are digging into the perceptual outcomes of the AB testing from headlines. Frankly, it provides context to the NYT’s own research and development of the algorithms and human processes that go into such efforts.

If the narrative is entirely that if we dont actively consider capturing interest, we’d be doddering and hard to track, if we do we’re abusive, then media is forever doomed to be unsatisfactory. We all hope to improve.

In the world of headlines, the “spiciness” that’s been advanced as a function of engagement hunting is something that’s currently contended with through human intervention. All headlines are human created and the outcomes of AB tests are more about improving the understanding between author/editor and captured audience than manipulation or future interest conditioning.

[disclosure: i lead ML platforms and the algo related eng products at NYT. all thoughts are my own presentation of what i have experienced, and not company opinions]

What we measure, determines what gets maximized (or minimized, if it's something we don't like). The factfulness of an article, or how well informed the reader is after they read it, are not easily measured. The clicks, are now easy to measure. What gets maximized, is the clicks.

It would be nice if we had a news source with a feature (opt in) where the reader got an email quiz the next day, asking just a few questions about the facts in the article. The feedback loop would tell us which articles actually make the reader better informed.

Emotionally charged, slanted or dramatic headlines is one of the reasons I cancelled my NYT subscription a couple of years ago. I'd picked up a student NYT subscription back then and was had FT subscription from school, reading them back to back makes their slant and editorialization pretty obvious even if it isn't the most in your face.

Correct me if am wrong but doesn't this mean they are eventually going to move entirely to a click-bait model of acquiring readers.

If this level of desperation is actually seen as acceptable for a media house, then journalism or whatever is left of it, is in dire need of help.

Great article. Stories also change under time and press engagement. Some of these may be feedback loop outcomes (cuomo) where others (spacex land explode story) i don't think are.

I would suggest that this can be more 'evil' that just trying to get more clicks.

A number of times I have seen on some place like Facebook where the initial article has some extreme headline, and then hours after when engagement is up, the headline is swapped to one that is less inflammatory. A few more extreme-perspective friends will send me an article saying "see?!?!" - and by the time I see it it's already been rewritten.

A few times I would click on an article going viral and find the headline in the article itself doesn't match the one cached by Facebook. Remember that most people aren't even reading the headlines and just assume that NYT are trustworthy. The first headline is the one they end up internalizing.

Bare in mind, there is zero consequence for doing this either. The newspaper and claim they were "correcting an editorial error", whilst openly spreading misinformation about some hot topic.

I think at the very least it should be mandatory to maintain a list of edits to an article once published - and to indicate to the reader clearly that the article has received a number of edits.

I’d agree with you but it seems like these alterations are pretty benign. They’re still professional and not at all fake/misinformation.

The entire thesis of the OP is that it is not benign and the intent is malice/inflammation rather than accurate summary of content.

It may not be "fake news" in the Fox srtain, but the OP's point is that it is not for the good of the reader or society.

If they consistently go from "max outrage to grab attention" to "more clinical, for posterity", that's not benign. It's dishonest. Either be a tabloid or don't be.

Having worked in A/B testing in a loan marketing company, where we were pushing 300% payday loans in Mexico (you read that right), I can talk of evil.

I think, when you're in the day to day of a web platform, you really just simplified everything from your hiring to your board meeting to one metric: click.

Everything is number of click, it becomes no more evil than a Lion in the Savanna, you just must increase your number of clicks somehow and you forgot why.

It can lead to interesting "let's remove the legal text away and see what gives", "let's color ever buttons blue vs pink and see who wins", let's reorder the search result independently of what a normal human would expect (as in... cheapest first?)".

I see what you mean, but it's not evil anymore. It's now part of nature, they must hunt your click. They can't stop and they can't explain why. They could put one headline "Myanmar military coup" then replace it with "Myanmar cabinet reshuffle" to see which one you click on with little input from any writer and no philosophical debate whatsoever.

Evil done by beauracracy, process, or algorithm, is still evil.

I know you weren't trying to suggest this, but A/B testing doesn't necessarily imply evilness, of course.

It's definitely evil in the parent commenter's and the article's cases, but if it's testing color schemes for your blog or a signup flow for your productivity app, there's a pretty good chance it's fine (depending on the underlying ethics of the blog/app).

But I overwhelmingly agree that ubiquitous or bureaucratized doesn't mean "not/no longer evil". If anything, there's possibly a light positive correlation. That comment's espousing one of the worst and most cynical philosophies I've ever encountered.

There might even be some natural/game theoretic phenomena at play here, too. Information wants to be free, and evil wants to be normalized.

Agreed.

Yes, there's a lot of good examples.

Here's one: https://nypost.com/2020/06/02/new-york-times-changes-headlin...

Before: “As Chaos Spreads, Trump Vows to ‘End It Now"

Edited: "Trump Threatens to Send Troops into States"

Another one:

https://thehill.com/homenews/media/489013-trump-rips-ny-time...

Before: "Democrats Block Action on $1.8 Trillion Stimulus"

After: "Democrats Block Action on Stimulus Plans, Seeking Worker Protections"

And changed yet again: "Partisan Divide Threatens Deal on Rescue Bill"

https://www.techdirt.com/articles/20200413/11142144293/lessi...

Before: "A Harvard Professor Doubles Down: If You Take Epstein's Money, Do It in Secret"

After: "What Are the Ethics of Taking Tainted Funds?"

To be fair, its not only the NYTimes that does this. One that was circulating around Twitter yesterday was: "How I got COVID after taking the vaccine" with a footnote in the story: "Author did not get COVID". The headline was changed to "Why could you still get COVID-19 after receiving a vaccine?"

NewsDiffs is a scraper that did this kind of automatic headline change tracking, but it hasn't been updated in a while.

https://github.com/ecprice/newsdiffs

This alternative looks more recent

https://github.com/DocNow/diffengine

The "Editing TheGrayLady" twitter account is the updated version that is still working

https://twitter.com/nyt_diff

Some comments on newly-released YouTube videos will restate the title of the video at the time the comment was made simply to annotate that fact for future viewers, and such comments are often highly upvoted.

An interesting reactionary adaptation to this "temporary sensational title" exploit.

Sounds like a plaster on a gaping wound. The comments engagement is quite low, even on very popular videos.

I think more needs to be done for publishing transparency.

The history of edits is of no use to the people whose minds have been poisoned with the original propaganda.

Yeah I somewhat agree, but transparency is surely a good start?

"And yet it rarely attracts the kind of close scrutiny of, say, a Fox News. And that’s totally reasonable! Fox News is an absurd clown show and deserves every criticism it gets"

I find it comical that any write-up critical of a left wing organization needs to include something like this to avoid being flagged, down-voted, whatever, on the platform its being posted on.

I don't usually read any US news as a rule, because of how sensational it all is. But I think I see most NYT headlines because someone inevitably posts them to NH. And honestly, when I compare them to what I saw on Fox News (which admittedly I only looked at for about a week around the election), I see no difference in the level of partisanship or spin. If anything there is less on Fox News just because the headline writers are not trying to show their undiscovered writing talent in the same way as NYT.

If excluded, I doubt anyone would have cared or complained.

CBC did this years ago.

Whenever I'd load the website, sometimes all the headlines would suddenly switch to different wordings, once sufficiently loaded.

The article is trying to paint NYT's A/B testing as nefarious, but unfortunately in the current social-media-oriented landscape, headlines are very, very important for clarity, and A/B testing headlines is something every media publisher with a sufficient tech infra does.

As frequent Hacker News submitters know, headlines alone can determine whether a post gets upvoted.

I think the article attempts to demonstrate with its examples that some of the titles don't actually accurately represent what ought to be the core takeaways of a news story.

Selected excerpts:

"I actually watched this interview—all two hours of it—and I can tell you that the first two headlines are a much better summary of what went down. Yes, Meghan does reveal that she contemplated suicide, but it’s a five-minute interlude in an interview that has a lot more going on."

"Trump starts off addressing conservatives and claiming leadership of G.O.P. but in the final headline Trump has a hit list and is firing a warning shot. And sure enough, the bombastic rhetoric propels this article onto the “most viewed” list."

In other words, while you're correct that titles are important for clarity, the claim in the article is that "clarity" can be at odds with what gets clicks (or gets upvoted).

I'm able to viscerally internalize this because, as a tech worker, I've typically used A/B testing as a tool to maximize engagement, but I've never in my career used it as a tool to maximize correctness or "clarity".

> headlines are very, very important for clarity

I'm pretty sure engagement is more important than clarity. I don't see how A/B testing helps you optimize for clarity. What are they measuring to determine a clarity score?

Sadly, most of the time engagement is highest when something is more outrage inducing.

> but unfortunately in the current social-media-oriented landscape, headlines are very, very important for clarity,

Clarity? Or click-thrus? It's definitely not the former. Which in turn discounts the NYT as a beacon for journalisms and associated standards. They're desperate for traffic. Clarity isn't a metric in the playbook.

Witin the world of high volume publishing, this type of A/B testing was popularized BuzzFeed.

Good piece, but damn I was hoping for some details on their implementation. Are the variations coming from the server, CDN/edge, or client? How does this impact SEO? Do they change the article slug if the title is a winner? How much is automated and how much is human curated? When beset with new assignments who has the time to babysit these things?

cdn; seo impact is also tested for; canonical URL doesn't change; titles are written by humans, testing is entirely automated; small parts of larger UX team just do testing so babysitting these is their job

+++ thank you! A stack to aspire to!

Is the headline testing some sort of multi-arm bandit system or a/b tests that are updated manually?

It seems like MAB would be the ideal approach for something like headlines.

How do you measure SEO impact in a title split test? Scraping SERPs or monitoring query volume real-time?

The value in A/B testing headlines is 30% boosting engagement directly and 70% creating a feedback loop where the newsroom can learn how to write a more engaging headline. Avoiding clickbait headlines requires some restraint and they are well aware of that slippery slope.

Feature request:

Service that scrapes websites known to A/B test things, caches the A/B test content, and presents all of the variants at once.

In this context, yes, I want all the headlines. Maybe they could be presented as a diff. With timestamps!

I think there is some bad intent at times - I think a reputable institution should be optimizing for accuracy and engagement, not sensation for its own sake. Different headline styles are suitable for different sections of the paper, as well.

I saw one the other day from Nick Kristof: "America is not designed for the anatomically correct". It was about the need for public restrooms, but the headline didn't make me read it. It sounded goofy and, other than knowing Kristof writes decent articles, I passed it over.

Later, the headline was "America isn't designed for those who pee". A bit coarse, but very much more clear as to what the topic of the editorial was.

Doesn't surprise me in the least. I've long felt that "journalism" has more in common with marketing and SEO than much else these days.

This is very reminiscent of how the YouTube algorithm when optimized for views and clicks drove a lot of people into deep rabbit holes.

Integrity and such aside, as someone who runs A/B tests I love seeing this data. Thanks for sharing it!

In several of these the shift is due to a headline written about an upcoming event being changed during and after the event, especially the Trump and Cuomo examples. Interesting tracking but not really A/B, which would be testing two different headlines roughly simultaneously.

I would say that learning how the NY Post writes their headlines would be a more fun story.

Reply on news.ycombinator.com