Ranking by Engagement | Hacker News Reader

Ranking by Engagement

tecunningham.github.io

77 pointsentrican12 comments

I would model things differently. In particular, I disagree with point #3:

"3. Engagement is negatively related to quality."

On Netflix, the two most watched TV seasons ever were Stranger Things Season 4 and Wednesday Season 1 [1]. Both were higher quality compared to "sensitive" shows like 365 Days.

Generally, I don't think engagement is negatively correlated to quality - rather, I think it's only negatively correlated (sometimes) when we restrict the domain to the most engaging content.

If you took random samples of YouTube uploads, I'm confident you'd see a positive correlation between quality and engagement, while simultaneously finding that the highest engagement videos are not the highest quality.

I believe I've heard this called splitting tails of the distribution, where if X and Y are correlated, the highest Ys are still unlikely to be the highest Xs.

I'd amend "engagement is negatively related to quality" as "the highest engagement items are not the highest quality items", which is a much narrower claim.

[1] https://www.whats-on-netflix.com/what-to-watch/most-watched-...

> I believe I've heard this called splitting tails of the distribution

Diverging tails [1], or Berkson's paradox [2]. One of the (imo) most underrated fallacies in statistics.

[1] https://slatestarcodex.com/2018/09/25/the-tails-coming-apart...

[2] https://en.wikipedia.org/wiki/Berkson%27s_paradox

Very interesting reads, thanks for sharing!

This seems heavily impacted by the specific proxy being used for "engagement".

In fact the article notes "In 2012 YouTube switched from maximizing clicks to maximizing watch-time. They found it led to a short-term decrease in clicks but a long-term increase in retention. I believe Netflix similarly has invested a lot of time in developing “deep” measures of engagement."

If you were to rewrite this article to not use the word engagement and say "user clicks" it would ring a lot more true, but the attempt to generalize from clicks to things like likes, etc, actually muddies the issue and impliesthis issue is more fundamental than it is.

In many ways, RL promises to solve this problem by letting us optimize metrics like user retention directly, even when there is not a way to decompose it into a per-item score, but RL is hard to deploy efficiently and has not seen huge takeup.

But to go back to the article, I think it's argument that estimating quality is important is clearly true, though often you can implicitly estimate quality from some other metric (e.g. aggregate watch time), but the article implicitly argues that you have to start from estimating clicks and everything else is downstream of that, which is not fundamentally true.

I think Netflix might be categorized as "other media" further down, where it says "social media platforms rank by predicted engagement ... ranking by popularity is common for other media", so the negative quality correlation might not apply.

Quality being negatively correlated with engagement sounds like social media version of people preferring junk food over gourmet.

You bring up a good point. When I read the post I applied a filter of "social media" so vetted everything from the perspective of Twitter, Facebook, Insta, etc. And a lot of the ideas hold up when compared to that.

It breaks down as you say with Netflix which probably means there's a line between user-generated content and curated content that requires a higher bar.

Loved Tom Cunningham's internal posts when I worked at Meta, great to see he has public blog!

Analyzing items on their own is as useless as analyzing single fireworks by color, size, and audience mouth gape. It's the dynamics of the whole fireworks show that matters.

It must be hard to say "quality is near to the primary goal of social media" and "nothing is wasted" in the same blogpost. You will always get trickling here, obfuscating there. Those who lack engagement or refuse to expose their habits are punished with lower quality buffet spread until they capitulate to the platform's definition of quality.

Say I wanted to rank my own personal collection of songs by retention/engagement— are there any open source libraries or crisp descriptions of algorithms/statistical models that one could use?

Six observations:

(1) Most platforms (FB, IG, Twitter, TikTok, YouTube, Netflix) rank content by predicted engagement: they'll show you the items most likely to cause you to click, reply, retweet, etc.

(2) Ranking by engagement increases retention. Users shown ranked feeds spend more time and return more often (higher DAU), compared to users shown unranked feeds. Platforms care about retention above all and thus they maximize engagement primarily as a means to that end.

(3) Ranking by engagement hurts quality. The most-engaging content often has high shares of clickbait, spam, scams, and misleading headlines. Low-quality content tends to hurt retention and so platforms are constantly trying to weed out engaging but low-quality content.

(4) Ranking by engagement increases "sensitive" content. It shows more nudity, bad language, abuse, hate speech, & partisan politics. In many cases decreasing sensitive content has a neutral or negative effect on user retention, so platforms are unsure what to do about it.

(5) Platforms would prefer to have less sensitive content (for a variety of reasons) but they also don't wish to be seen to be making value judgments. As a consequence they often are slow to intervene, both on quality and sensitivity, and they intervene in roundabout ways.

(6) Incorporating explicit user preference typically doesn't make a big effect on sensitive content. E.g. if you ask users survey questions they often seem to like the sensitive content.

"Engagement" can mean completely different measurements. Clicking headlines is a measure of initiating attention, video watch time is a measure of maintaining attention. Maintaining attention seems like a great measurement of the relative quality of various videos.

Platforms ban content depending on their sensitivity instead of the quality of discussion. It makes it hard to have high quality discussions about sensitive / contoversial topics (which may not be important / hard to target for ads).

Reply on news.ycombinator.com