Where can I get a data dump of all HN comments?

9 points11235813217 comments

I'm looking for a data dump of HN comments. It looks like Hacker News API[0] had one and disabled it. I haven't been able to find any other. It doesn't really need to be up to date -- anything in the last ~3 years would be fine.

[0] http://api.ihackernews.com/

View on HN

Instead of scraping HN directly, have you looked at the HN Search API, from the old HN search engine ? ( https://www.hnsearch.com/api )

The current search engine looks like it also has an API ( https://hn.algolia.com/api ). Presumably they have a full dataset on hand, which you may be able to get if you contact them.

Scrape it

10/10 would scrape

Surely there's a better way than hitting the HN servers O(7169040) times...

Does anyone have an old scrape they can post somewhere or as a torrent? Note, for my purposes bag-of-words (i.e. word counts) would be enough.

bag-of-words (i.e. word counts) would be enough

Are you doing sentiment analysis?

Sorry for the delay in replying: I didn't see your comment.

I'm working on topic modelling (Latent Dirichlet Allocation in particular). It's for a business, but I also want to solve one of my problems: reading HN addiction ;)

I think you just suggested that scraping HN was a constant time operation.

Reply on news.ycombinator.com