Being a long time reader of HN I'm sometimes frustrated that top HN content is often coming from popular websites or talk too much about the same thing (Zoom security issues, Facebook leaks, ...) I wanted to understand if this was possible to get only the 'original content' posts of HN (from the not-as-popular blogs and sites). Because to me this is the most interesting part of HN.
So I started analyzing HN new posts, and made a few discoveries :
- Some HN users post a lot of content, and post several links in a row, which will push away your post if they publish just after you
- You get a 30 min time window at busy hours (and 1h time window at non busy hours) between the time a new link is posted, and the link disappear from the 1st page of new links
- There is a second-chance pool for good stuff if a moderator detects it
As a result HN is overflowed with not really useful content and it is not always easy for original content to be noticed (Even if I think HN is doing a very good job compared to any other link aggregator that you can find)
So I tried to built a tool to filter out things like news websites, words that I want to blacklist, and users whose posts haven't been relevant to me.
That way I'm able to remove almost 80% of content and I can go through the list of all the links of the day before going to bed
For those who are interested, this is just a cron job querying the HN API every 3 minutes inserting the new links into a db, and a web server rendering the last 500 links.
You can see more on how the filters works here: http://hn.luap.info/about and you can also understand which links have been filtered here: http://hn.luap.info/links_flagged