Hacker News Reader

Comment by dilyevsky | Hacker News Reader

Imo these tiny quotas are specifically put in bc a lot of people (especially on this site) always complain how they managed to rack a huge bill by accident. You can’t have it both ways

View on HN

Well maybe if they implemented reasonable ways to cap your bill then they wouldn't need have to have all these arbitrary hidden limits. It's ridiculously difficult bordering on impossible to make sure that a usage spike or bug won't cause an infinitely large bill, on any service they offer. App Engine used to be the exception with an actual spending limit, but (of course) they deprecated and removed it.

The latency to query cost is on the order of hours on user side so the service would be off the limit by the cost rate * cost latency. That’s my guess for why it’s not a feature.

I doubt there is a fundamental limit which says the reporting needs to be that slow. I suspect that some metrics are slower to query than others, but it would go a long way if you could get near instantaneous billing wherever possible. Even better if you could have a hard shut off if cpu/disk/network spend goes over x limit. I choose to believe: it is easy money if people cannot easily avoid accidental mega bills.

See my other reply. The reason I believe it’s architected this way is to have a good tradeoff between timeliness and overhead cost. Full correctness at global scale would involve serializing all billable events through a series of global transactions, which would bring performance to a crawl (e.g., 10ms just to synchronize the metric).

This is absolutely right, and I among others have asked AWS to especially please not put any checks on performance "inline" just to do capacity limits. We are happy with the present tradeoffs.

The needs of those wanting to compute should absolutely outweigh the needs of those wanting to not compute.

There is no fundamental reason why they can't do better here. There's no law of physics preventing accurate billing caps. And to the extent that they implement their own caps imperfectly the sensible approach would account for the extra cost as simply part of their costs of running the service which should be budgeted for in aggregate and covered by the pricing that they set themselves. It's simply not a priority for them, clearly.

There actually are some fundamental laws related to this problem. It’s a distributed system, so you can lose availability of the metrics data and be bounded by the CAP theorem. You’d also need to keep the metrics synchronized at some granularity across zones, effectively serializing all billable events at a global scale. For example, if you wrote one byte into storage across the globe, that write would have to be billed, committed, and be globally visible before the next byte could be written and billed. And each communication is bounded by speed of light, etc. You’d effectively be synchronizing all billable events through a database at cloud scale.

Sure, in theory absolute perfection is difficult. In practice the achievable level of accuracy within the laws of physics is well within the bounds of reasonable and far, far beyond what is implemented today. You don't need absolute perfection, since as I suggested you can simply allocate some budget to cover overages amortized over your entire customer base. Especially since the vast majority of the time you'll be operating in a regime where you are far away from the billing cap, so that gives you a lot of leeway. There would never be a need to synchronize every byte written to storage.

I agree you can get something working. However, if 1% of the time, a customer is overcharged $100+ because they assumed the limit was guaranteed, you’d probably reconsider if you want to offer that service at all.

If 1% of the time a customer uses $100 extra, you don't charge that customer $100 extra. You charge them $0 extra because they had a bill cap and it was your job to enforce that cap. Then you raise the price of the service a very small amount so customers pay $1 more on average and that covers everyone's overages (that's what I mean by amortizing across your customer base). Then you go and improve your limit enforcement because you are actually incentivized to do so, unlike in the case where customers are charged for overages that aren't their fault, where you are actually incentivized to cause overages. Customers aren't dumb, they can see your incentives (see the comment by 0cf8612b2e1e in the adjacent thread).

It’s plausible to do what you’re saying. I still think it’s more trouble than it’s worth. If you somehow managed to engineer everything perfectly and tune it as you say, you’d still have customers who wanted their service to stay online past overcharge. I’d think the predominant business case for such tight guarantees would be small and low budget projects. Additionally, every cloud vendor who doesn’t do this seems to be cheaper, so you’d be priced out (in a commodity market).

> you’d still have customers who wanted their service to stay online past overcharge

Then they wouldn't configure overcharge limit, would they?

I mean, why not? I have alerts on cost in my cloud usage corresponding to orders of magnitude increases in expected cost. If they trigger a few hours faster (more accurately) that’s good. But I don’t see what I would do bar shutting down the service, so my argument is that it’s not really a feature that would make a difference for most use cases. And if your service is steady, you can already do a linear extrapolation from yesterday’s costs to see if you’d go over budget.

Except they're in an adversarial relationship with malicious users so as soon as they open up any bit of slack, they open up an attack vector for people to steal their compute resources.

I maintain that it is possible within the laws of physics to do a good enough job enforcing quotas that losses are not high, with minimal runtime overhead. The problem is that cloud providers are not incentivized to invest in accuracy, in fact the opposite. Anyway, truly malicious repeat offending customers can be dealt with in other ways.

Isn't main reason that when there are no limits people spend more money?

You can't have what both ways, specifically? User-facing spending and capacity limits that are hard to accidentally set too high, without also unilaterally and without warning terminating compute nodes for opaque, alleged ToS violations when their compute actually gets used?

The system approves requests by how trustworthy you are. It does this so: 1. You can’t denial of service the cloud or something else 2. Launder money through mining cryptocurrency 3. Take popular instances away from high profit customers

This isnt why. Its because these services cant scale to everyone having unlimited quotas

If you have quota that doesn’t prevent stock out at all which cloud support will be happy to tell you when you have tons of quota but can’t spin up any more VMs in a region

No, but it provides some control. They will stock out if everyone suddenly uses their entire quota (they aren't reservations) but by limiting the quotas that they issue they can reduce the impact of single actors or large swings of smaller actors on availability.

Basically it improves the predictability of resource usages by limiting outlier spikes.

Seems like it'd be simple to deal with that:

Allow people to pre-pay (deposit) an amount of their choice and set an account flag "Do NOT exceed account balance".

You will have to do transactional billing at cloud scale which isn’t really feasible

no way. it’s because they can’t scale to unknown demand.

Having quota does not guarantee you get the resources. Commit is supposed to (but not really)

I hate this excuse. AWS, GCP etc are not designed for children. They are designed for serious, enterprise-level workloads.

People need to do their due diligence.

You are going to need a serious, enterprise billing account attached to it to have quotas automatically increased.

They wouldn’t call it clown compute if it was serious

Reply on news.ycombinator.com