Tell HN: Hacker News BigQuery dataset is updated daily again

51 points·wietsevenema··11 comments

As you might know, there's a Hacker News BigQuery dataset [1].

I recently noticed that the export was broken (since Nov 2022) and I worked with the team to help fix it. It'll now again run daily.

SELECT timestamp FROM `bigquery-public-data.hacker_news.full` ORDER BY id DESC LIMIT 1 # 2024-06-09 09:12:55 UTC

[1]: https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news

View on HN

Daily is too infrequently.

I have this dataset being updated in ClickHouse in real time: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...

I also provide a way to export it or attach it to clickhouse-local and analyze it locally: https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...

I love that this dataset is accessible via cors from web frontends. Got me into experiments with clickhouse

Same with github, much better than than throttled apis

Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.

Any interest in getting this dataset on Snowflake too?

(I run a Snowflake funded data provider on Snowflake marketplace)

The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API), so anyone is free to do this.

a bit off topic: is there a way to display a HN page using the firebase API/sdk without doing N+1 requests ?

(sorry for noob question)

No, unfortunately. We're going to make a new API that will be much easier, but there's at least one other major project that needs to get done first.

Thanks for the response!

Yes, that’s how it works. Same data as in the API.

How big is it?

Closing in on 41 million items

Reply on news.ycombinator.com