Ask HN: Has Hacker News stopped uploading its dataset in 2022? | Hacker News Reader

Ask HN: Has Hacker News stopped uploading its dataset in 2022?

console.cloud.google.com

10 pointswslh13 comments

That doesn't appear to be an official HN thing, it's some third party responsible for it. I see dang has said he contacted them but can't do anything about it. The API still works.

https://news.ycombinator.com/item?id=38796861

I now understand that. I just thought it was trivial for HN to export the public database buit if not it can be retrieved from the API. Are there hard rate limits to use the API? And we can assume that everyone has a license tu publish the HN public data? In general, there is a disclaimer about that. Don't want to be sued though.

You can now get Hacker News data in real time from the Hacker News API powered by Firebase: https://github.com/HackerNews/API

This is great (real time!), but also kind of a pain (38+ millions individual http requests to get the whole thing).

Thankfully there's no authentication or apparent rate limiting. I fumbled my way through downloading the whole thing with curl. I screwed up a few times so made over 70 million requests in total.

Toy analysis of the data I downloaded here: https://public.tableau.com/app/profile/isna/viz/HackerNewsDa...

After one day of my post and reviewing these answers a simple query [0] gives a lot of responses. For example [1] and [2].

[0] Google: dump all hackernews site:github.com

[1] https://github.com/ashish01/hn-data-dumps

[2] https://github.com/iOliverNguyen/hackernews-dump

The graveyard that is Google's issue tracker has an abandoned ticket about this: https://issuetracker.google.com/issues/261579123

Email this stuff to the mods or search the site

https://news.ycombinator.com/item?id=38781031

Check https://news.ycombinator.com/item?id=38781031

There is no [official] answer there. Dang [1]?

[1] https://news.ycombinator.com/user?id=dang

https://news.ycombinator.com/item?id=38796861

If it is a matter of [human] resources I can put engineers to work on this as an open data good.

You might have missed it in all the links but the data set and its maintenance is a Google thing not an HN thing so those are the people you need to get in touch with.

So I assume your feedback is official right? I understand now that it could be an interesting contribution to have a similar database to be used, for example, for machine learning? Do you agree?

"random internet busybody" official, yes. If you want to talk to HN people about it, you can email them, I'm just pointing out their description of the situation is already quite plain and public.

Reply on news.ycombinator.com