Has anybody else had this experience? Or are we just doing it wrong?
This is not intended to be a rant, just curious.
The best way I've found to approach it is to treat GCP as something that has to be evaluated at an individual service level. It's great if you're on one of their expected workflows/golden paths, and you can get lucky with a good fit if you aren't, but they seem to have a lot of unspoken assumptions and limits baked in that might or might not align with your use case.
Disclaimer: My use cases are pretty unusual from talking to our account rep, so this might be over-fitting to weird data.
It was originally estimated like 10k or something per test which was approved at the time (had like 3 level of management all down my neck for getting it out, hence using lambda originally).
We did deliver, just needed one more sprint to rewrite it as a distributed system on servers. ;) Moved to like 20 machines w/ 128gb of ram that we could spin up as needed (testing millions of events a second, system in NodeJS!)
My record is $20k and it raised some eyebrows. But it was not really a mistake, just a sub-optimal backfill.
The data was filling a need for making appropriate business decisions, and compared to all the money lost by business developers making investments on a hunch, this was a very small bump in the road.
But really soon I noticed the slow startup … simple queries took too long (1.2 sec vs milliseconds in a traditional database)
Then I learned a lot about BigQuery views. That helped a little.
At some point I simply wanted to export data. New Google tools needed to be learned: Cloud Storage, Data Flow.
After 18 months of using BigQuery on roughly 850 million rows, I switched back to a traditional database.
I’ve never used Pub/Sub or Cloud Run, but have been quite happy with BigQuery and GKE.
When you put a query in the BigQuery console, it'll tell you "This query will process ??? MB when run" at the top right.
So if you code all your queries interactively in production (which is what everyone else is doing anyway) it's not too hard to keep an eye on.
Note that this is not the default! :-)
We found that all of these had significant caveats that required careful planning. We had a few instances of runaway AWS costs due to basically not knowing enough about AWS and we had to be careful to only use the "good" AWS products, Digital Ocean never had runaway costs but they did keep turning off production services because our use-case was not one they were familiar with (dev machines, off-site backups). Bare metal was a minefield, we found we couldn't reliably run Prometheus because it ate SSDs. As for GCP, it did require understanding the pricing and it was possible to shoot yourself in the foot with things, but no more than anything else.
There are going to be gotchas everywhere. Overall we had a great experience with GCP, to the point that the company has remained on GCP post-acquisition by another company who were mostly on Azure.
But I do agree, there are some gotchas. PubSub examples: Duplicated messages, shitty DLQ implementation (in my opinion), some developers had improper error handling which lead to to infinite resends of messages (because they nacked message on error), etc..
However, I think the scaling and setup weighs up for all of that. You just need to specify a topic and subscription, and then you don't really have to care about resources or scaling at all, and that is SUPER nice. Also, PubSub is stupidly cheap in comparison to any other similar product, at least that I know of.