Comment by tveita | Hacker News Reader

Comment by tveita | Hacker News Reader

It's a pain. I recently had a compute job that would take a couple of days to run locally. "This is exactly what the cloud is for," I thought and went to Google Cloud where I already had an account.

First try I'm allowed to spec up a beefy server with AVX-512 and click "create", but it never actually starts. Turns out the default CPU limits are tiny - I need to apply to bump two separate limits and wait for that to go through.

Second try I'm actually able to spin up a couple of servers. I set the compute job running and go for lunch. An hour later both are terminated and I get a message about "violation of our Acceptable Use Policy".

There is a link that says crypto mining is forbidden on free instances. I send a reply informing them that a) these are not free instances, and b) I am not mining crypto.

I get a reply back saying more or less "Yes it says free instances but why would you expect anything we write in the documentation to be correct. Also you might think you weren't mining crypto but actually your VM was probably compromised because no one would buy compute instances and actually run compute jobs on them. We're not giving you back access to your instances, maybe delete them and create new ones lol."

I actually find it an easier product to get started with than AWS, but man they really really don't want your money. When you are actively trying to prevent your customers from scaling up resources you are failing at being a cloud.

Imo these tiny quotas are specifically put in bc a lot of people (especially on this site) always complain how they managed to rack a huge bill by accident. You can’t have it both ways

I hate this excuse. AWS, GCP etc are not designed for children. They are designed for serious, enterprise-level workloads.

People need to do their due diligence.

You are going to need a serious, enterprise billing account attached to it to have quotas automatically increased.

They wouldn’t call it clown compute if it was serious

You can't have what both ways, specifically? User-facing spending and capacity limits that are hard to accidentally set too high, without also unilaterally and without warning terminating compute nodes for opaque, alleged ToS violations when their compute actually gets used?

This isnt why. Its because these services cant scale to everyone having unlimited quotas

If you have quota that doesn’t prevent stock out at all which cloud support will be happy to tell you when you have tons of quota but can’t spin up any more VMs in a region

No, but it provides some control. They will stock out if everyone suddenly uses their entire quota (they aren't reservations) but by limiting the quotas that they issue they can reduce the impact of single actors or large swings of smaller actors on availability.

Basically it improves the predictability of resource usages by limiting outlier spikes.

Well maybe if they implemented reasonable ways to cap your bill then they wouldn't need have to have all these arbitrary hidden limits. It's ridiculously difficult bordering on impossible to make sure that a usage spike or bug won't cause an infinitely large bill, on any service they offer. App Engine used to be the exception with an actual spending limit, but (of course) they deprecated and removed it.

The latency to query cost is on the order of hours on user side so the service would be off the limit by the cost rate * cost latency. That’s my guess for why it’s not a feature.

I doubt there is a fundamental limit which says the reporting needs to be that slow. I suspect that some metrics are slower to query than others, but it would go a long way if you could get near instantaneous billing wherever possible. Even better if you could have a hard shut off if cpu/disk/network spend goes over x limit. I choose to believe: it is easy money if people cannot easily avoid accidental mega bills.

See my other reply. The reason I believe it’s architected this way is to have a good tradeoff between timeliness and overhead cost. Full correctness at global scale would involve serializing all billable events through a series of global transactions, which would bring performance to a crawl (e.g., 10ms just to synchronize the metric).

This is absolutely right, and I among others have asked AWS to especially please not put any checks on performance "inline" just to do capacity limits. We are happy with the present tradeoffs.

The needs of those wanting to compute should absolutely outweigh the needs of those wanting to not compute.

There is no fundamental reason why they can't do better here. There's no law of physics preventing accurate billing caps. And to the extent that they implement their own caps imperfectly the sensible approach would account for the extra cost as simply part of their costs of running the service which should be budgeted for in aggregate and covered by the pricing that they set themselves. It's simply not a priority for them, clearly.

There actually are some fundamental laws related to this problem. It’s a distributed system, so you can lose availability of the metrics data and be bounded by the CAP theorem. You’d also need to keep the metrics synchronized at some granularity across zones, effectively serializing all billable events at a global scale. For example, if you wrote one byte into storage across the globe, that write would have to be billed, committed, and be globally visible before the next byte could be written and billed. And each communication is bounded by speed of light, etc. You’d effectively be synchronizing all billable events through a database at cloud scale.

Sure, in theory absolute perfection is difficult. In practice the achievable level of accuracy within the laws of physics is well within the bounds of reasonable and far, far beyond what is implemented today. You don't need absolute perfection, since as I suggested you can simply allocate some budget to cover overages amortized over your entire customer base. Especially since the vast majority of the time you'll be operating in a regime where you are far away from the billing cap, so that gives you a lot of leeway. There would never be a need to synchronize every byte written to storage.

I agree you can get something working. However, if 1% of the time, a customer is overcharged $100+ because they assumed the limit was guaranteed, you’d probably reconsider if you want to offer that service at all.

If 1% of the time a customer uses $100 extra, you don't charge that customer $100 extra. You charge them $0 extra because they had a bill cap and it was your job to enforce that cap. Then you raise the price of the service a very small amount so customers pay $1 more on average and that covers everyone's overages (that's what I mean by amortizing across your customer base). Then you go and improve your limit enforcement because you are actually incentivized to do so, unlike in the case where customers are charged for overages that aren't their fault, where you are actually incentivized to cause overages. Customers aren't dumb, they can see your incentives (see the comment by 0cf8612b2e1e in the adjacent thread).

It’s plausible to do what you’re saying. I still think it’s more trouble than it’s worth. If you somehow managed to engineer everything perfectly and tune it as you say, you’d still have customers who wanted their service to stay online past overcharge. I’d think the predominant business case for such tight guarantees would be small and low budget projects. Additionally, every cloud vendor who doesn’t do this seems to be cheaper, so you’d be priced out (in a commodity market).

> you’d still have customers who wanted their service to stay online past overcharge

Then they wouldn't configure overcharge limit, would they?

I mean, why not? I have alerts on cost in my cloud usage corresponding to orders of magnitude increases in expected cost. If they trigger a few hours faster (more accurately) that’s good. But I don’t see what I would do bar shutting down the service, so my argument is that it’s not really a feature that would make a difference for most use cases. And if your service is steady, you can already do a linear extrapolation from yesterday’s costs to see if you’d go over budget.

Except they're in an adversarial relationship with malicious users so as soon as they open up any bit of slack, they open up an attack vector for people to steal their compute resources.

I maintain that it is possible within the laws of physics to do a good enough job enforcing quotas that losses are not high, with minimal runtime overhead. The problem is that cloud providers are not incentivized to invest in accuracy, in fact the opposite. Anyway, truly malicious repeat offending customers can be dealt with in other ways.

Isn't main reason that when there are no limits people spend more money?

no way. it’s because they can’t scale to unknown demand.

Having quota does not guarantee you get the resources. Commit is supposed to (but not really)

The system approves requests by how trustworthy you are. It does this so: 1. You can’t denial of service the cloud or something else 2. Launder money through mining cryptocurrency 3. Take popular instances away from high profit customers

Seems like it'd be simple to deal with that:

Allow people to pre-pay (deposit) an amount of their choice and set an account flag "Do NOT exceed account balance".

You will have to do transactional billing at cloud scale which isn’t really feasible

I am curious why would they care if you mine crypto or not? If you pay for an instance, why not?

If they don’t fight that use case, I suspect they become a lightning rod for stolen credit card transactions.

It's bad for the rest of us customers. a) more effort goes into fraud, more risk of annoying false positives, b) easy to max out resource utilization which shortens hardware lifespan, c) boom and bust industry makes planning harder which means they can't run on as predictably low margins. This all increases costs and makes free tiers harder to offer. Miners are an easy group to say no to because they don't overlap much with the rest of the software business world, so the reputation hit is small.

$20 says 100% of crypto mining in cloud = your customer got p0wned

Oh, thats an easy $20.

I mined Ethereum in azure during the height of the crypto boom, and looking at my statistics of vms i was able to get i am pretty sure some other people did to.

On 5k$ spend on azure i got round about 7k$ in Etherum when sold the next day, this includes german Sales tax and some loses i made starting out when not accounting for the german sales tax early on.

I think the whole timeline where this worked was about one and a half weeks(Germany, if you played around with exchange rates and sales tax potentially a bit longer), but it worked pretty well.

Azure was also perfectly fine with me doing it.

Yeah, and once that brief period of easy arbitrage was over, 100% of all people who are paying $10 to mine $1 worth of crypto are people who are trying to steal the $10.

Reply on news.ycombinator.com