‘Reverse Engineering’ Hacker News into an API | Hacker News Reader

‘Reverse Engineering’ Hacker News into an API

lukakerr.github.io

52 pointsrashadphil16 comments

The main issue with all apps like this is not the reverse-engineering, which is relatively straightforward, but the fact that HN sits behind Cloudflare, with the WAF and ratelimiting enabled.

It's not possible to achieve capability-parity just *reading* HN as the API does not return comment karma, which is used to sort the view. (I think this is the subtler half of why HN reader sites never catch on - the interesting comments are never at the top, so the presented content is very noticeably that much more boring.)

So a while back I tried building a simple scraper component (an internal API, I guess) that would use Firebase for realtime updates, and strategically poll the site (say one request every 3-5 seconds) to get the current sort order. I never got to the Firebase part because I crashed into Cloudflare while fiddling around tuning the ratelimiting.

So yeah. You need full Chrome (to satisfy whatever Cloudflare's WAF throws at you) and a captcha-solving subscription (because the WAF will get the flyswat out when it sees your datacenter IP) to be able to do this properly - especially if you want to APIify this sort of thing.

I recently wrote a HN reader and decided to forgo handling comments for reasons you mention, including the obscurity of comment karma scores. I focused on the story browsing side of things, and the links to comment sections take you straight to HN’s comment page.

Link to my app if anyone’s curious. Red question mark in upper left corner of page goes to its “about” page:

https://www.thnr.net

Actually if I recall correctly, even though official api doesn’t show Karma value of comment, the children comment IDs (kids is the field name) of each comment are actually sorted by karma (or whatever method HN uses to sort on server side)

I tried creating a GraphQL API[0] for HN for both read and write operations so I could eventually use it to write a HN client app but I hit a wall due to HN scraping policy. It limits you to fetching a page once every 30 seconds[1]. Yes, I confirmed with dang that I would need to adhere to this.

[0] https://github.com/searchableguy/hackernews_graphql

[0] https://hngraphql.fly.dev/graphql

[1] https://news.ycombinator.com/

Replicating the features that require being logged in such as upvoting and commenting is actually not trivial.

I recently built a Hacker News client [0] on top of the official API and using the method described in the article to login users is very inconsistent. I kept running into Captcha issues and even when that didn’t happen, sometimes a login request would simply not return the expected token.

Unless the official API starts offering user login, this will remain difficult.

0. https://modernorange.io

For years, i just want to be able to easily search through my "favorite submissions" which I use as a kind of bookmark. Guess we will never see that. Not on the website nor in an app.

I actually setup my personal website to track my favorite submissions. https://rashad.vercel.app/hackernews. Favorite submissions can be made public and are easily scrape-able. What do you mean by easily search through?

Did you try writing to the HN mod(s) asking about that? (At the address given on the guidelines page) Sounds like a good idea.

(2018)

Bottom of article says August 2nd, 2017 even

Didn’t HN get an official API a long time ago? Or was that read-only?

Link at the bottom of HackerNews that says “API” points to this GitHub repository [1].

The first commit is from Oct 2, 2014 while the article was published on April 1, 2018.

However, it seems that you did not read the first paragraph of the article which states the following:

> While building hackd I faced a problem - the official Hacker News API doesn’t allow for interaction, such as upvoting, posting and commenting. I wanted hackd to be a full featured Hacker News client, so this wasn’t going to cut it.

So to answer your question:

> Didn’t HN get an official API a long time ago? Or was that read-only?

Yes, it appears that at the time of publication, the API was read-only.

[1] https://github.com/HackerNews/API

What do you mean "at the time of publication"? It's still read-only, unless I'm missing something obvious.

Search hackd in iOS app store

No results. First two results are "Jack'd" and "Grindr", followed by many more dating apps.

Reply on news.ycombinator.com