So far the best solution I've seen to save on BigQuery costs is to deploy your own ClickHouse.
It has its own gotchas and quirks, but cost-wise it's not even comparable.
It has its own gotchas and quirks, but cost-wise it's not even comparable.
Thank you! Glad it helped you.
ClickHouse is in particular well-optimized to take on real-time analytical workloads from BigQuery - both functionally and from a cost perspective.
For anyone wanting to try it yourself, there guide to follow when syncing or migrating workloads from BigQuery to ClickHouse, with some benchmarks and back-of-the-envelope cost comparisons: https://clickhouse.com/blog/clickhouse-bigquery-migrating-da...
There is a place for BQ but it is good to set expectations correctly and also look at the constraints. They are sometimes not obvious. The docs do helpfully outline the limitations, for example:
- Materialized views: https://cloud.google.com/bigquery/docs/materialized-views-in... - Indexes: https://cloud.google.com/bigquery/docs/search-intro#limitati...