Hacker News Reader

Comment by minimaxir | Hacker News Reader

How does this differ from the Algolia HN API in terms of data access? (https://hn.algolia.com/api) I was able to download all HN data recently with ease using that endpoint. Authentication?

EDIT: After looking at the documentation there are two new aspects of the Firebase API not in the Algolia API:

1) Ability to see deleted/dead stories.

2) Endpoint for user data.

Question to kogir/dang: Has the "delay" field (Delay in minutes between a comment's creation and its visibility to other users) always been there?

View on HN

The 'delay' field has been there for years. I vaguely remember pg announcing it here when he added it, so we could probably dig up the exact date.

> "Delay" field post by pg on 2008-06-29 https://news.ycombinator.com/item?id=231024

I'm also curious whether it removes some of the limitations of the Algolia version; I wanted to download my content for some statistical analysis (notes at http://www.gwern.net/HN ) but I discovered that it seems there's some hard limits to how much of my data I can reach: https://github.com/algolia/hn-search/pull/36

There's a cheat to work around that limit: use the created_at_i parameter.

Example: https://github.com/minimaxir/hacker-news-download-all-storie...

If you want to get data for a single user through Algolia's API via the commandline, you could also use https://github.com/jaredsohn/hnuserdownload. It uses the same technique as minimaxir's code (a post of his was the inspiration.)

I don't know Python, so I'm not sure what your source code is doing. At a guess, you've hacked together some sort of repeated queries thing with a time-window?

> Get 1000 entries and process them. > Take the timestamp of the last entry. > Requery the API, asking for articles made before that timestamp

Repeat.

Algolia scrapes HN HTML and provides their own API from what it looks like.

They don't scrape it.

From the hnsearch about page:

"Every 2 minutes, the last 1000 HN items (stories, comments, polls) are sent to Algolia's indexing API. Items from the last 48 hours are refreshed every hour."

Correct, and we're planning to move to the new official API ASAP (instead of the legacy one -> it was not web crawler but far from perfect) for the indexing.

Regarding the REST APIs, let's keep both for now :)

Reply on news.ycombinator.com