Comment by mritchie712 | Hacker News Reader

Comment by mritchie712 | Hacker News Reader

Parent Full thread

> they're saving over 60% off of their BigQuery bill

how big is their data?

A lot of BigQuery users would be surprised to find they don't need BigQuery.

This[0] post (written by founding engineer of BigQuery) has a bit of hyperbole, but this part is inline with my experience:

> A couple of years ago I did an analysis of BigQuery queries, looking at customers spending more than $1000 / year. 90% of queries processed less than 100 MB of data. I sliced this a number of different ways to make sure it wasn’t just a couple of customers who ran a ton of queries skewing the results. I also cut out metadata-only queries, which are a small subset of queries in BigQuery that don’t need to read any data at all. You have to go pretty high on the percentile range until you get into the gigabytes, and there are very few queries that run in the terabyte range.

We're[1] built on duckdb and I couldn't be happier about it. Insanely easy to get started with, runs locally and client-side in WASM, great language features.

0 - https://motherduck.com/blog/big-data-is-dead/

1 - https://www.definite.app/

They have >1PB of data to ETL, with some queries hitting 450TB of pure shuffle.

It's very true that most users don't need something like BigQuery or Snowflake. That's why some startups have come up to save Snowflake cost by "simply" putting a postgres instance in front of it!

In fact, I just advised someone recently to simply use Postgres instead of BigQuery since they had <1TB and their queries weren't super intensive.

> A lot of BigQuery users would be surprised to find they don't need BigQuery.

No they wouldn't.

a) BigQuery is the only managed, supported solution on GCP for SQL based analytical workloads. And they are using it because they started with GCP and then chose BigQuery.

b) I have supported hundreds of Data Scientists over the years using Spark and it is nothing like BigQuery. You need to have much more awareness of how it all fits together because it is sitting on a JVM that when exposed to memory pressure will do a full GC and kill the executor. When this happens at best your workload gets significantly slower and at worst your job fails.

Hopefully, we can be another managed solution for those on GCP.

And as for your second point, yep, Spark tuning is definitely annoying! BigQuery is a lot more than jusr the engine, and building a simple interface for a complicated, high-performance process is hard. That's a big reason why I made ParaQuery.

You may want to look into DataMechanics who is another YC startup who tried something similar. They were acqui-hired by NetApp.

If I remember they focused on SME space because in enterprise you will likely struggle against pre-allocated cloud spend budgets which lock companies into just using GCP services. I've worked at a dozen enterprise companies now and every one had this.

Enterprises can deploy on their own GCP, and we're planning on releasing on GCP Marketplace.

For a similar cost, what if their pipeline were 5x faster, and they don't have to dealing with managing the deployment themselves?

Thanks for telling me about DataMechanics

a) Enterprises have almost entirely moved away from self-hosting software. GCP Marketplace is fine but I would probably also look at a Kubernetes option as many companies have GKE clusters.

b) It won't be 5x faster though and I wrongly recommend you don't take a marketing attitude when selling this type of software. Because it will be mostly technical engineers and architects deciding on this and we aren't stupid. I have run GPU accelerated Spark clusters for years for enterprise companies and you will be able to accelerate the query part of the pipeline but that's like 20% of what a typical job does.

a) Since being fully-managed is one of my value props, that's probably better for us.

b) Of course I'm only accelerating the Spark/query part. Not sure what you mean. And in that case, I took a query which was 44 minutes on BigQuery and ran it with a "comparable" cluster on ParaQuery in 5.5 minutes. Perf is slightly variable, so maybe it's 40 minutes vs 6 minutes. In that case, ParaQuery would still be 6.5x faster, and >2x cheaper. That being said, it was just a benchmark ETL query with some random data (50b rows), and these things do vary between workloads.

So yeah, without knowing more about the use case you're talking about, hard to say. Even Nvidia has a hard time optimizing certain TPS-DS queries btw, so it's not like I can just 5x anything!

> No they wouldn't.

haha, you're giving people way too much credit. Tons of people make bad software purchasing decisions. It's hard, people make mistakes.

Reply on news.ycombinator.com