Hacker News Reader

Comment by summerlight | Hacker News Reader

I frequently see this kind of surprising billing anecdotes across many cloud providers. Why don't they provide a way to set a hard budget limit applied for the entire account. I tried to see what can be done for GCP and this seems pretty daunting.

https://medium.com/@steffenjanbrouwer/how-to-set-a-hard-paym...

View on HN

The reasons are probably quite complicated, because some of them are bound by hard technical limits to how quickly a system can react and thus make a hard limit actually a hard limit, but realistically that's largely solvable just by making it a softer hard limit (eg you set a limit of $1000 and the terms say you pay that plus whatever is used before the limit kicks in. More that $1000 but way less than $14000).

All of those technical reasons aside though, the commercial reason is obvious - people's mistakes and overages are a great source of revenue and profit. Companies refund the times where it'd be enough to lose the customer, or when it hits HN, but they make more money every time someone pays up. They have no incentive to fix it. It's part of the business model.

There are no conceivable "hard technical limits" that make such a system difficult. It's 100% commercial.

oh there are - billing systems at scale almost exclusively work on logs. Logs can take minutes or hours to aggregate and transmit to a central place.

Ever notice how your "1GB" data plan sometimes lets you use 5GB if you happen to be roaming in another country and downloading something fast over 5G...? Same reason.

They are also checking your account .. and as easy as you can lock an account, as easy you can soft lock it via a flag because the billing system says enough. And the few cents in between they swallow easily (as with so many other inaccuracies).

They do not want it, that is the only reason.

There is also the fact that if a company has critical systems go down because GCP hit some hard budget limit, it will be reported in the press as "Netflix down globally due to issue with Google Cloud".

Google doesn't want the bad press. Most real companies would prefer to have a big bill when their product surges in popularity than have unexpected downtime at the worst time.

> Most real companies would prefer to have a big bill when their product surges in popularity than have unexpected downtime at the worst time.

That doesn't preclude it being an option.

It’s really not part of the business model.

Sure, this guy fat-fingering $10k sounds amazing.

But GCP deals with businesses paying for years of service. Multi million dollar deals are common.

Google and AWS and the like could give a flying fuck about anything under $100k.

Because then we'd see articles about how the next start up missed their opportunity whenever their site unexpectedly got discussed on the latest Rogan episode and subsequently was taken offline by the limits being tripped.

Companies could make the limit optional and pass 100% of that downside to the customer. 99.9% of customers would opt in.

I don't see the problem. Don't set a budget limit if you don't want your app to go offline. Lots of people wouldn't mind if their app went offline for a bit. They'd prefer to not suddenly get a $10,000 bill

There's no "right" answer. In one case, it's checked the wrong box and got a $14K bill. In the other case, it's I checked the wrong box and my startup missed its one window. There are in-between levels of alerting etc. for both populations but they're probably unsatisfactory for the extreme conditions.

To be clear: I'd be very in favor of the major cloud providers having a "DO NOT! DO NOT! use this for production mode and your content could be deleted at any time if you screw up. But I suspect most people wouldn't use that."

Google AppEngine used to have that but — presumably in the interest of additional profit — they removed it. Now I have to make do with an alert that warns me long after I could be hypothetically bankrupted, i.e. in seconds.

I learned of that billing limiting mechanism after the $14k was charged to my account. As designed.

What incentive to cloud providers have to give you that ability? I think they greatly appreciate the ability to accidentally spend a lot of money

To prevent a bad PR like this? When it goes viral, most customer supports escalate it to a higher level then they just eventually cancel the bill.

Google does not care about bad PR like this. It doesn't affect their biggest customers.

What bad PR? This is just a kid who can't read and thinks they can process 2PB of data for absolutely free. No GCP customer will care.

tbh, I have worked with AWS for at least 10 years, and recently their field support are quite prone to help avoid those scenarios (e.g. helped to save hundreds of thousands in a single-digit million account).

This was one of the main selling points for all portfolio companies of the group to adopt AWS in their digital transformation projects.

Limited use for a nobody who wants to run <$100 / year cloud spend and does not have account managers.

I would love to kick the tires on some AWS stuff, but the threat of unlimited ruin is not worth it. Sure, maybe the gods would take pity on me and wipe the debt, but far easier to just run with someone who caps costs. My toy project can gladly go down if the alternative is a huge unexpected bill.

An unhappy customer won't come back.

The OP is probably a good person with strong interest in data science and building projects.

If it'd be "oh here's your $500 charge, upgrade your quota for more, 'ok fair enough, I did a mistake'", but $14k is not ok without explicit quota upgrade.

> An unhappy customer won't come back.

Unfortunately, if the customer has written their applications in such a way that they're effectively locked to the platform... they won't have much choice until they can dis-entangle themselves.

After that though, yeah. ;)

My cynical self sees it as how cloud providers aim to make the most money: by making billing oblique and waiting for buzzword-happy project leads to mandate stuff be put on their service without understanding what the end billing will be.

I can't say that's for certain what it is. I just know a hallmark of any business with recurring charges that are otherwise incomprehensible is so they can hit you with the charge after the fact, and you have little recourse to avoid paying it without a ton of work for yourself or your team.

Because they aren't sufficiently incentivized to make giving them your money harder.

The problem is that part of your bill may include things that cost money even if nothing is "running".

Would you like Amazon to delete all your files, disks and backups once you hit your limit?

Also Static IPs, Load balancers, DNS zones?

Google does provide a way, project owners can set a custom quota to limit costs.

So your comment made me go look it up, and if you squint hard that's kind of true...

https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

Notice that their "solution" is to tell you how if you want you can spin up effectively your own custom service to watch spend and if it goes over some threshold delete the entire project[0] after some delay. This is the malicious compliance version of letting you add a limit.

[0] At least, that's how I interpret "This example removes Cloud Billing from your project, shutting down all resources. Resources might not shut down gracefully, and might be irretrievably deleted. There is no graceful recovery if you disable Cloud Billing. You can re-enable Cloud Billing, but there is no guarantee of service recovery and manual configuration is required."

I meant this, linked directly off the big query cost estimation docs:

https://cloud.google.com/bigquery/docs/custom-quotas

> Custom quota is proactive, so you can't run an 11 TB query if you have a 10 TB quota. Creating a custom quota on query data lets you control costs at the project level or at the user level.

Oh, good catch! Yes, that does look like something that can be coerced into limiting it. Having actually tried to click through, it is very much not as simple as "don't spend more than $X"; the doc points to https://console.cloud.google.com/iam-admin/quotas and you have to find and set the right quota, but yes that can probably help.

Isn't it the quota limit that you need to set? https://cloud.google.com/bigquery/quotas#query_jobs

the setup of the budget limit isn't complcated. the linked article goes thru putting the monitors/alerts on pubsub and etc. which isn't mandatory.

Reply on news.ycombinator.com