Improving the Hacker News Ranking Algorithm

I'd be happy to see some experiments with ranking algorithms, maybe in a separate site, or as options within HN as someone else suggested. There are some downsides to both, though (potentially low exposure for the first, technical complexity for the second).

What worries me is the definition of quality you use. We look for submissions that we find valuable for us, not necessarily high quality. Interests are varied, and we all get value from different things. Quality might be very correlated with value, but it's not quite the same. And here comes the big issue: maybe more than 50% of the value we derive from HN comes from the comments. I feel many times we are upvoting submissions by sheer relevance, so we can have valuable discussions about a relevant topic. I don't think the given metrics are capturing this. I like the analysis and the proposal, but it's easy to see how there's some very important perspective missing.

> [I]magine two submissions with the same number of upvotes and a different number of views. The article with more views probably has a lower quality than the one with fewer views, because more people have viewed it without giving an upvote. So the ratio of upvotes and views is a signal of quality.

I believe this creates a bias against longer articles. If a submission links to a longer piece, a user could take a long time to come back to HN to upvote the submission.

That's not to say this would be any worse than the current algorithm. By definition, a time-ranked frontpage and moving discussion will always favor shorter articles, or, if the content is long, produce plenty of comments that are only superficially related to the article.

As a submission ages, its rank wanes, and discussion gets more sparse: a piece that takes time to read, understand and digest will probably perform worse than a short one, since the discussion and votes will take place further into the future.

Excellent analysis and I think the proposed algorithm would work well against clickbait titles. HN could trial several different algorithms over a period of a week each and watch the metrics. Thinking out loud:

Part of HN’s charm is that niche content can surface to the top and many here are experts in their field, contributing great insights unavailable in normal media (example: pilots discussing possible causes after an airplane crash). That means the rest of us get to learn something new with perspectives from the best and many newbies become interested in more. That’s what top University classrooms feel like. I’m not sure what metric would quantify that quality (just # of comments as a metric encourages rage posting, maybe # comments with high upvotes?). The current system, although game-able and full of false negatives also has entropy built in - randomness that is more likely to give a wider range of content to the front page. That helps with genetic algorithms, so it might be of benefit, but missed if not quantified. If gaming the system counts - the most persistent also win some more.

Another option to consider is duration of stay on articles. From testing it seems Google factors in duration of stay on content as search criteria as well, which skews rankings towards longer pages. Those also tend to be ad-supported and likely within Google’s own advertising network. Within HN, more comments increase duration of stay, so that would likely increase provocative content, so may not be a good idea, but testing would surface that and more.

Have you tested the behaviour of your algorithm on pagination and dynamics of link position over a day?

I assume the good thing about the snowball effect is that "making it to the frontpage" gives you guaranteed and pronounced exposure over a day or two - even though it's not maximizing the number of good links shown.

Sounds good to me. I've had very little engagements (good or bad) on things I've submitted in the past that I thought should have been interesting to at least a few people. It seems there is a lot of content falling through that just isn't timed right to get somebody to push the up vote button. I had 1 upvote on a link I submitted last weekend. I suspect nobody actually saw the link or clicked on it and that this was not based on any merit. I've also had links submitted with no action only to see the same link submitted by somebody else making it to the front page much later.

Duplicate links are actually interesting since they are so easy to detect (identical link). Why not simply aggregate metrics for those things? The important thing is the link making it to the front page without creating a lot of duplicates. Somebody submitting a duplicate would effectively become an up vote for the already submitted link. Views and duplicate submissions are possibly more significant signals than up votes.

In search precision and recall are the two metrics people use for judging search quality. It's important to realize that hacker news is effectively a ranking algorithm and therefore is a search engine; even though it delegates actual search to Algolia. It sacrifices recall for precision: everything on the front page should be relatively high quality. But that's at the price of potentially high quality things never reaching it (recall).

Of course, with only 30 slots on the front page, there's only so much that can be on it. Especially if you consider that many users only drop by once or twice a day or so. So those slots stay occupied for quite long as well. Days in some cases. The choice as to what is right is highly subjective (i.e. the moderators decide) and biased towards the intentions of the site and that's intentional. But that doesn't mean it can't be improved.

How about you let HN users choose what algo they want like how they choose their header Color.

> The low scores are a bit surprising because all submissions got enough votes to make it to the front-page.

The entire premise of this section is incorrect. There is no such thing as "enough votes to make it the front-page". There's no single threshold, nor is the value of a single vote constant. You're going to have a lot of other effects like specific domains having score penalties, votes being ignored due to them appearing to be fraudulent / non-organic, or the votes arriving too diffuse over a long time period. (Duplicates of the same URL will count as a vote for something like 12 hours after submission, so it's quite easy for a link to get a long tail of votes even after dropping out from the first page of /new.)

I would bet that most of what they claimed made it to the front-page never did. There used to be some HN stat-tracking sites around that had the full ranking history of submissions. Joining against that data would be a lot more credible.

> To achieve this goal, the new-page should expose every submission to a certain amount of views, to estimate its eligibility for the front-page.

That is obviously unworkable. /new is a slush pile, 90% of the submissions do not deserve even a single view based on the title. The proposal ensures that people visiting /new will only see the obvious garbage that nobody else has clicked on either.

Are 'views' defined anywhere in this writeup? I couldn't quite figure out what the added ranking parameter 'views' exactly is and how it's measured.

I noticed later submissions often had higher scores. Could this be a function of hn getting more popular over time?

I have been thinking it would interesting to do the following experiment. Pick two random new posts and show them directly on the home page for a fixed amount of time. Feed the data from those posts (views, votes, content, comments) into some neural network which can magically learn what kind of posts generate most interest.

Showing random, new posts on the home page provides equal viewership to the experiment set while removing the age bias. Many candidate posts will likely be low quality, but we can expect people to filter them out and participate less in them. High quality posts will organically attract people's attention and the algorithm can over time learn factors that differ between lower quality and higher quality posts.

Does the assumption of a homogeneous pool of content and a corresponding homogeneous pool of readers affect this? Segmenting by topic and applying the algorithm per each pool might be necessary if one wants to have a "fair" assessment of quality.

If you want the staff to see this, email them at the contact link below.

An obligatory link to an article by Evan Miller about ranking: https://www.evanmiller.org/how-not-to-sort-by-average-rating...

However, to get the following requirement from the article:

> The algorithm should not produce false negatives, the community should find all high-quality content.

it might be better to estimate the upper confidence bound, like in Upper Confidence Bound bandit, rather than the lower confidence bound.

One of these days, I am going to write a custom front end.

The biggest problem I have with the front page these days is it gets clogged with junk science and low-effort news posts that produce entirely predictable flame wars that just aren't very interesting. There's about two dozen domains that account for 95% of such stories. I go through and flag/hide them now, and that works okay, but doing it automatically would be better.

Also, the ranking algorithm should take into account the rank of the content when the upvote happened - 10 upvotes for a content on the 5th page is much more impressive than 10 upvotes for a content on the 1st page (keeping everything else constant).

I would love to test-drive this via a webapp. Any plans like it?

There are situations where some posts come to front page hours after going out from the new page

It has happened to me a couple of times as well. That is clearly unexplainable.

I typically browse via my rss reader / feed, so even though I’m here daily, I never see the front page (I do upvote on comment pages however).