Hacker News Reader

Comment by deepsun | Hacker News Reader

BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.

View on HN

Thanks for that tip, this is exactly how I would do this if I had to do it from scratch. Just in case it's useful for anyone else: https://docs.cloud.google.com/bigquery/public-data

The crawlers would have still just hammered their site though, right?

I think the idea is that they could store the data in BigQuery, and point users of the site there.

The crawlers would have still just hammered their site though, right?

Why is your comment exactly word for word of another comment just one level above in the comment chain?

Because the crawlers would still have hammered their site, though.

(The GP post doesn’t actually meaningfully address the issue being raised. Adding BigQuery or whatever would not change the fact that (a) they already offered a method of getting all of the data in a cost effective way, and (b) the issue was that the crawlers hammered the site hard enough to make it economically unviable.)

Maybe because they restated what they said instead of addressing to the previous commenter’s point.

The comment I’m speaking of was from a different user, and it’s giving a strong smell that both users are bots unfortunately. Maybe I’m wrong, maybe it’s a coincidence those exact strings of characters were formed in isolation from each other in the same thread. I hope I’m wrong, but it’s a red flag

I'm not a bot, and the other commenter was restating the question as the solution provided did not seem to actually solve the problem, just redesign the entire project (See comment from u/taneq).

We didn't form "those exact strings of characters in isolation from each other in the same thread" they are directly referencing my comment in a reply thread to said comment.

Because I am pointing out that the same exact thing would still happen even when commenter's "fix" would be applied.

You're bad at understanding context. Word for word was just me poking fun at that comment

Sure, but now you're moving the site from "Free data presented in a pleasant way to view" to a "pay-as-you-go database". Your audience shifts dramatically, and you lose the ability to share the data you're trying to present.

True. But it sounds like they already lost that.

No, it's free for everyone for the data sizes they have:

Free tier: 10 GB of active storage and 1 TiB of query data processed per month.

There is also the deep magic...

https://github.com/phiresky/sql.js-httpvfs

I'm curious how this compares to just using DuckDB in the browser?

https://duckdb.org/2021/10/29/duckdb-wasm

Different pros and cons -- SQLite is smaller is the big one

Yes, if I wanted to put a nice HTML interface over the query results then I would still end up with the same problem where some combinatoric explosion of query parameters to the `/search` endpoint, most of which are cache misses, leads to many many TBs of network egress.

I think snowflake has similar, you can rent it out or make it free

Reply on news.ycombinator.com