Hacker News Reader

Comment by minimaxir | Hacker News Reader

A reminder that BigQuery (as used in the query in this link) is the best way to play with Hacker News data; don't scrape HN data manually!

The `bigquery-public-data.hacker_news.full` table appears to be up to date with the most recent HN data as well (table last updated today).

However, I'm not 100% sure the query is correct for unilaterally getting all links, as running the query on the full dataset returns the same results as running it from 2006-2015. And I value my sanity enough to not fuss around with the regex.

View on HN

What is the best way to download this dataset? Last time I messed with it I had to pay for a Google Cloud bucket and run through some awkward sequence of steps to eventually get a local copy.

I think I ended up following the advice here: https://stackoverflow.com/questions/18493533/how-to-download....

That's essentially it (export the BQ table as a CSV to a Google Cloud Storage bucket, then download it from there), but you can do that entirely in the web UI, no CLI needed.

If you just want a subset of the data, run a query, then save the query as a table in your project and export from there.

Does it cost money to do that or it is only time consuming?

It’s not free: Only storage and egress, which are trivial.

Running the query in BQ is free up to 1 TB and the 16000 row download is free so I recommend that if necessary.

I think an alternative option could be to have a torrent (or other file sharing mechanism) with the public HN information. Am I missing something? 16k rows seems very tiny for doing an analysis of HN.

The vast majority of potential HN analysis can be done in SQL alone within BigQuery, which is the proper way to handle a data warehouse regardless.

You mostly need the raw data for ML model training which is a very niche use case.

You can read it directly into a pandas DataFrame. [1]

https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

Oh no way, thank you!

This pandas trick is for small result datasets, in which case the 16,000 row limit from BigQuery more than satisfies it.

I would not recommend using it for the full many-million-row dataset.

Reply on news.ycombinator.com