If I pay $39/month, a default cap should be $39 per-month. Otherwise, let me set a cap I am comfortable with.
Surprise billing is never good for customers, only the business.
We agree completely about surprise billing.
If it were critical infrastructure, or monetized in a way that brought in revenue to cover the charge, then maybe I don't want it to shut down despite skyrocketing costs, but that's hardly the only situation you could be in.
Secondly, the best solution is to simply stop everything. But now the customer has to cold-start their entire infrastructure, which may actually cost more than paying the bill.
Thirdly, it is likely that customers will set a billing limit and then forget about it years later. Suddenly, they've got a complicated infrastructure setup spanning the globe. They finally hit a scale where they hit their billing limit that they had completely forgotten that Bill configured in their early days (who doesn't even work there anymore). Suddenly, the entire global infrastructure is shut down in the middle of the night.
That's the gist of what my friends said.
https://www.forbes.com/sites/sergeiklebnikov/2020/06/17/20-y...
Although even then, "we will only threaten to charge you $100k without meaning it" isn't much of a reassurance.
There is also usually some "soft" limit Y < X that you don't want to exceed, and don't plan to exceed, but you'd rather pay >Y than face an outage.
But a hard limit would have to be set to X to avoid that outage, and if it gets exceeded, you'll face a bill of X and an outage.
So what a customer would actually need is to specify both X and Y, with the rule: If the cost would exceed X, then terminate it early so the cost doesn't actually exceed Y.
Sounds complicated to implement, but then, the current practice of waiving the bill is complicated too if you tried to formalize it.
(For the sake of this discussion, I'm ignoring all the technical difficulties of terminating a high-availability service at all.)
I definitely get the idea of: I don’t want to spend X so if it looks like I will, terminate service at Y. But I think that’s a special case of the general situation, I want to know how much I’m on track to spend, right?
But I don’t know much about this at all. My whole experience was accidentally getting my own personal self a $500 AWS charge and then deciding they cloud services were dumb.
I don't know. I just tried to frame the problem from a customer's point of view, because cloud vendors' statement that customers would not like a limit is (IMHO) limited by their POV. Customers do want a limit, but not the way that cloud vendors would implement it. I think a huge part of the problem is understanding what exactly it is that you need when you use a cloud service. (This is varying from customer to customer, and from service to service, of course. You usually have important services that must be running, and others where an outage would be unpleasant but not critical.)
> Nobody wants to spend any money really right? But you need to spend some to avoid losing service. That’s just a cost you don’t like but need to pay.
That is not the issue. From a customer's POV, I would be ready to spend extra to keep the service running, but there is a limit where I'd prefer an outage because I can't bear that much. There are two problems with that: First, the limit is blurry. Second, a simple hard limit would leave me with a huge bill AND an outage. I would want to be able to choose one of those evils, not be left with both. And these two problems compound.
> But I don’t know much about this at all. My whole experience was accidentally getting my own personal self a $500 AWS charge and then deciding they cloud services were dumb.
I don't think they are dumber than the alternative. If you run your own hardware, you have a hard limit in both cost and computing power. You could technically get that with the cloud too, but it is not usually offered because it doesn't really solve the problem, but neither does it for for your own hardware.
That said, it would be nice if the major clouds would offer a "hard limit" option, but it really only works for "unimportant" applications that are cost-sensitive and can take an outage.
Of course cloud vendors do put themselves in the customer's position, and that's why they say that customers would not be happy with a limit, even though they are asking for it.
But to extend that to the billing case, you’d have to have a partnership with your customers, not just a dashboard where they push buttons and an API where you add or delete machines.
Maybe the website goes read only except for admin traffic when the budget is exceeded, for instance. Not as a bespoke process each company has to reinvent, but as functionality provided by the vendor.
It's different here, sure, but the providers optimize for not letting customers shoot themselves in the foot, and remediation via bill forgiveness is a fine solution -- from the provider POV.
A company isn't liable if their customer gets themselves hacked because they decided to not use any of the many MFA options available to them and neither is a company liable if the customer set a billing limit rule that they executed correctly.
Companies can simply not be trusted to tell the difference between a foot-gun and a..whatever a good kind of gun would be...
> aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text | xargs -n1 aws ec2 stop-instances --instance-ids
Will stop all ec2 instances.
The real fix is scoping credentials on aws - if you don’t use an account or role with limited permissions then even if they had this toggle the first step in an attack would be to disable this option.
Reversibly (i.e. shut down compute, don't delete anything, allow the customer to review, fix and reinstate quickly) terminating service is a minor annoyance for hobby/experimental setups, and in those, it's much more preferable than having to open a support ticket to deal with a massive bill.
Having quotas that the customer can increase themselves (but has to manually choose to increase) on storage prevents storage related surprise bills, and the rest you can shut down (optionally, make the user choose up front what they would prefer).
What am I missing? Too many commercial customers picking "experimental" initially and forgetting to change it?
I would also expect a price cap feature to be useful for experimental/no-approval-required projects at work. In fact, if I ran a cloud project for work as a small team-internal project, a cost explosion would become an even bigger bureaucratic nightmare than if it happened at home.
The problem is 100% technical. Detecting unexpected charges, scaling and restriction in real time is hard.
It's easier to just charge people money than come up with good ways to avoid charging them, and deal with edge cases as a manual process.
Sure. I get it. What company has an internal team that's like "ooo... lets find ways to cap the amount of money people pay us".
No one.
That's why.
> there are reasons other than malice and avarice
Right.
It's just avarice. There's no other reason.
> It's just avarice. There's no other reason.
The problem is technically solvable, it’s good for customers.
All you need to do is prioritise actually doing it.
In what stupid world do we pretend this is somehow good for consumers? Racking up thousands of dollars in bills?
It’s just the status quo.
They’re not not fixing it because it’s hard; that’s just the convenient excuse; they’re not fixing it because they don’t want to.
We have been measuring CPU, MEM with extreme granularity, how about considering price as a resource and measuring the same way, so that a service with a price cap can self manage and self terminate according to some priority field?
This might not be the actual solution, but we have been at this for a very long time, seems like there is not even a hint of an attempt at solving it by the giants. This is about incentives, sorry.
It made me feel much more comfortable hacking around and not needing to worry that I'd accidentally create a render loop or something that could rack up a bill whilst I wasn't looking
This would still allow all production services to run, but would stop someone from spinning up 200 crypto miners. I'm sure AWS is capable of implementing this, and I don't want to say it's "easy" but I would be shocked if they lacked the technical expertise to do this.
If you look at lots of these threads you’ll see that many people don’t want to provide phone numbers, lots of people ignore emails, even repeated ones, directly to them, from billing.
This isn’t a technical problem, it’s a service problem. I can see the hn posts already “my site went viral and <HOST> shut me down”
When you're paying $50,000/month for something that generates $200,000/month in value, or if an outage can generate $100,000/month in costs, or if the people that can fix an outage cost $100,000/year, then it's not.
That eventually will be factor into the price like credit card fraud insurance. Better have it be more transparent.
All of these incur costs. How hard a cap do you want?
I'm not talking it down. Maybe people are right about this. We'll see.
- a soft alert limit, which you set to the threshold of “hmm something is wrong but we’ll bear the cost until we figure it out”
- a hard limit which fails until more tokens trickle back in, without shutting down service
I really don’t want to rely on forgiveness, it’s just encouraging reckless behavior and submitting to the incomprehensibility of cloud pricing.
Everyone wants these limits, why not design products with that in mind from the get go? It feels like such an afterthought