Hacker News Reader

Yes we understand taking issue with the word fair. But we should say we mean fair in a very specific way. We would say our algorithm is more fair in the sense that it, in some ways, it more fairly reflects the intent of the HN community as revealed by their upvote behavior. We talk about this more in the Readme: https://github.com/social-protocols/news#readme

I don't agree that "fair" is inherently subjective. There are many sensible ways to objectively define fair. For example you could say HN ranking is "fair" if the probability any viewer would upvote that article is independent of its position-history on HN. That is an objective definition that is "fair" with respect to positions.

This and other notions of "fairness" are very common problems in ranking (I used to rank things on Instagram) that have to be addressed, even if you're only doing content based ranking.

"Fair" is inherently subjective because, in a vacuum, everyone has a subjective definition of the word. It can legitimately mean many different things, after all. People will choose the definition they read it as according to their own subjective experiences.

I reacted poorly to the use of the word "fair" here, too, because I didn't see how "fairness" really entered into it. Naturally, you can provide a specific definition of the word to make it objectively measurable, but if you use a word before you've said what your definition is, people are going to use the common definition -- and therefore, it's subjective.

The word "is" is also subjective in this context, since "subjective" is not a binary.

> It is time that we turn our back in the collaborative filtering algorithms that made social media a hellscape and embrace content-based filtering.

Except people deeply care about what other people are doing.

That was the whole point of Google’s Pagerank algorithm.

So, it might not be what you personally want. But to a lot of people, it’s more important to read/consume something popular (ie. that a lot of others care about), rather than something related to their own interests.

A good system uses both but it's not trivial to blend them. My system is right now showing me maybe 30% of what it ingests, if I was seeing just 3% I'd have to cut back more harshly and a popularity score would help. Fundamentally a popularity score has a much larger dynamic range than a relevance score.

Google has both a document-query relevance score plus a document quality score.

I've heard from a lot of people who like reading HN from a comment-centric point of view and I tried feeding all the comments into my system and it was really too much. When I fed in high-scoring comments, however, I like the results. I had somebody suggest comments from Metafilter and I think that could be a winner but of course comments have a network structure of relatedness to other comments and the submission that a comment-oriented reader could take advantage of.

  >So, it might not be what you personally want. But to a lot of people, it’s more important to read/consume something popular...

Count me in the "don't care" group. Which is why I always browse HN by "New" [0]. I neither know nor care which stories have been voted to prominence on the default frontpage, nor how this ranking has been determined. I just want to see what's new and select for myself what looks interesting.

[0] https://news.ycombinator.com/newest

I would suggest that one of the reasons you find the submissions on the New page valuable is that submitters are actively seeking out stories they think the HN community would upvote. So the content of the New page indirectly reflects popularity within the HN community.

Not really. There's a hell of a lot of wading through junk, when you browse HN by "New". So much so that I maintain a special uBlock filter list to weed out a lot of crappy domains and keywords.

That's one advantage of browsing by the default frontpage: your fellow HNers have pre-filtered what ends up there, for you. It's also the disadvantage.

Further in community like HN, it’s not just about collaborative filtering to find things that you as an individual will like. It is about focusing the collective attention of the community on a small set of topics to drive rich discussions.

it is well calibrated and just today my RSS reader told me a story I thought was a nothingburger would succeed on both metrics and… It did!

I was originally going to joke that maybe you should turn your script on to the stock market, but I'm guessing with your background you may have some experience in that regard!

I've tried that and failed but I was using a crappy commercial sentiment analysis engine.

These guys succeeded though and wrote a great book about it

https://www.amazon.com/Trading-Sentiment-Power-Markets-Finan...

That sounds like the sort of thing I'd enjoy reading - thanks!

Just looked up your profile. There's some super interesting stuff you worked on. We'll get in touch!