FYI, the new Hacker News API allows easy programmatic access of story/comments and infinite chronological paging. You could download every Hacker News story in less than 2 hours without breaking the API request limit.
[1]: https://chrome.google.com/webstore/detail/hacker-news-sideba...
Also, you can set hitsPerPage = 1000. ;)
"you can only fetch the 1000 hits for this query, contact us to increase the limit"
It was a nice try, but it did seem too good to be true.
Just pass created_at_i<X, where X is the time stamp of the earliest submission.
I was able to download 500k stories (i.e. about half of HN's 1.26M stories) before I ran into memory issues; I've fixed them and am downloading the rest.
- Python module: https://github.com/karan/HackerNewsAPI
- REST API: https://github.com/karan/HNify
RATE LIMITS
We are limiting the number of API requests from a single IP to 1000 per hour.
If you or your application has been blacklisted and you think there has been
an error, please contact us.You can query 1,000 stories per request, and do 1,000 requests per hour. That's 1M requests per hour. There are 1.26M Hacker News stories indexed by the API. :)
EDIT: Finished downloading all the entries (and can confirm that 1.26M is indeed all of them). Took 3 hours due to a conservative wait period between each request to make sure I stayed within the limits.