Hacker News Reader

You just need to make 38 million requests to the Firebase API document here https://github.com/HackerNews/API. :-)

I actually made many more than that because I screwed up a few times. Might clean up the compiled data and put it on Kaggle (GitHub? Torrent?), or anywhere else if people have suggestions.

View on HN

I thought you were joking about the 38 million requests! wow! Have you heard about .parquet files? Save it as a parquet and share the link.

https://www.kaggle.com/datasets/nickalt/hacker-news

this is amazing -- thanks. side note: when you want to get more data you can just get the new data, and save it as hacker-news-part-2.parquet - then when you use duckdb / malloy you can read them as a glob (hacker-news*.parquet) - and it is if you were reading one file.

Great tip. I used this little project as an exercise to some of learn DuckDB and Parquet. Still thinking about the best tradeoff for file format. An extra complicating factor is that items can change after they are posted so to get the most recent data, it's a good idea to re-request at least some range of historical items.

I'd definitely put it on Kaggle

Good idea. I put it here: https://www.kaggle.com/datasets/nickalt/hacker-news

Reply on news.ycombinator.com