AWS bill capping feature request thread still unanswered after 10 years
forums.aws.amazon.com
forums.aws.amazon.com
I believe GCP currently has the best feature for this. You can set both billing alerts as well as caps. However, I also believe that it can take up to 24 hours between incurring a cost and it showing up on your billing report. So even on GCP (which is the most forward-thinking cloud service for this feature), you can incur up to 24 hours of charges over your maximum billing threshold cutoff. I'm also not 100% sure if GCP's billing threshold is really designed to be a "hard cap" per se.
The real question is whether AWS should let perfect be the enemy of good; and/or whether providing a somewhat "broken" service like GCP's would mislead customers into feeling more protected than they actually are.
See here where someone set a Firebase billing budget of $7 but an infinite recursion generated $72,000 in charges. When the founders started seeing the charges come in, all they could do was watch as it grew and grew....because their screen was merely reflecting what had already happened in hours past.
https://www.theregister.com/2020/12/10/google_cloud_over_run...
Discussed here: https://news.ycombinator.com/item?id=25398148
And if the costs become exorbitant, Amazon is in a better position to improve their own systems to reduce the amount of overages that people run into in practice.
In theory, Amazon's first leadership principle is Customer Obsession. (See https://www.amazon.jobs/en/principles for the full list.) If they took that seriously, then setting this issue to rest for their customers would be a no-brainer.
And if you call amazon support and talk to them, especially as an individual, you might get your bill cancelled.
The "might" in your post is doing a lot of work.
FWIW I know of a startup whose video sharing app was used to reshare a pay per view football match and they incurred a $30k bandwidth bill that AWS did not cancel. That killed the startup. It was largely their own fault for not securing the platform well enough, or moderating popular streams, but being able to cap their AWS bill would have kept them in business..
Though, apparently population both caring about it and avoiding Amazon as result would pay less than cost of implementing it and not refunded income from catastrophic runaways.
AWS does indeed obsess about the customer. Every step along the chain there is someone there advocating for the customer. There are mechanisms to keep the customer in mind even for the developers who actually code the service and don't talk to customers on a daily basis.
I've had many, many service team members shadow me as I worked their service's tickets. This is explicitly so they can see in real time customer pain-points. If a customer has a question about a unique use case, the service team will proactively reach out to support engineers to set up a call to discuss the use case further. There are monthly (or twice-monthly) meetings between support service owners (i.e. those people in support who 'own' a service) and service teams to identify the top issues customers are having with the service. AWS is constantly looking for ways to better assist customers, make support less difficult for customers, increase self-service options for customers, etc.
I'm really, really curious where the basis behind your argument. Because from everything I've seen and been a part of, it's simply untrue.
I guess 'customer obsession' would be giving away everything for free?
Customer obsession would be NOT shipping buggy, unreliable software like AWS Amplify.
Customer obsession would be CloudFormation-first.
Customer obsession would be not forcing me to upgrade to a paid Support account to report a bug.
The list goes on, unfortunately. I do believe AWS employees mean what they say, but the external reality (IMO) is it takes a lot of time and effort to get AWS to notice their customers unless you're one of the big boys.
Customer obsession means not shipping bugs? OK, Bob, let's see your code.
CloudFormation is wonderful and essential. But AWS clearly optimizes for delivery speed. SOME service customers want Cloudformation from day 1. Others would rather have the API first, and have CFN a few months later.
Hence the "ask on account creation". If I want a dev account, I can choose to cap it. The amount of SMB that would benefit from this is staggering.
> Customer obsession means not shipping bugs? OK, Bob, let's see your code.
A major difference being that my company isn't worth $1T+.
I've used AWS for a long time and spoken at length with many wonderful, intelligent people in the company; and I didn't mean to tread on anyone's toes, I just wanted to express how it feels as a customer who spent >£150k/month for half a decade.
I feel dirty defending AWS, but this is one case I'd give them the benefit of the doubt. There must be _a_ reason they haven't implemented this yet and that reason must be somehow protecting the customer. "Customers want this" ends the discussion around here. You must have a really good reason to disagree.
Like $35 bank overdraft fees protect the customer from not getting a candy bar.
Oh I didn't think I was starting 1000 instances of miners in the EC2 GPU cloud, it certainly exceeded my configured budget, please give my money back..
There are many solutions to this problem to prevent abuse. All of which are way better for consumers than the status quo of everybody having an $unlimited spending limit.
If starting 1000 instances exceeds your configured budget, they could simply not start them, and shut down whatever number of instances you managed to start as soon as their cost exceeds the budget.
But least all Amazon accounts are tied to a credit card, so abusing in a scale similar to e.g. the GitHub case is not that easy.
All AWS accounts start off without the ability to do this (via the quota system) and being able to start 1000 ec2 instances of any type is a setting that needs to be unlocked via a support request (which can never be done by that support person, but always needs to be escalated to the "service team" and takes about 1 business day).
This is a billing question, not a technical question, and looked at through that lens it's easy to put a hard limit on a monthly bill: just don't ever issue bills greater than that amount. If I say I only want to pay a maximum of $1000 a month, and I hit that limit but it takes a bit for the provider to shut everything down so really $1100 of resources were consumed, then the provider eats the $100 overrun and I get a bill for $1000.
With an actual hard limit you create a financial incentive for the provider to minimize this overrun. Yes it might be difficult to fix but I assure you, if hard limits existed, the technical issues would be solved soon enough because now there's a reason to invest in a solution.
I can easily believe it, and I can easily believe that AWS employees have heard such stories, but I'd love to have it be more than an anecdote.
That raging customer might well assume that because almost all limits are like that, all are, including the new S3 limit, but the S3 limit causes service degradation forever, not during peak load. The writes that failed for a while map to reads that'll fail forever, because that data isn't there.
We can come up with possibilities that sound more or plausible. I'd love to hear something more factual.
https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
The short answer was 'we can, but don't want to' (note : this may be completely unrelated to what Google thinks internally, and is just what the fairly high up the food chain rep told us)
its on the "total spend" of a billing account level though, and obviously you'd have to be a billing administrator, so to work with it would is awkward; many billing accounts across disparate projects is basically the only way.
It's impossible to solve perfectly with tech.
Amazon saying "If you put a number in this form we won't charge you more than that, but your account will be limited by <list of limitations> and if you go over those limits then <long list of conditions that will apply if you go over the limits up to and including removing you from the platform if AWS think it was fraudulent>" is a perfect solution.
The only reason why there would need to be a perfect tech solution is if AWS are concerned about giving people a small amount of service that they're not able to charge for. AWS clearly believe protecting themselves from overages is more important than giving customers peace of mind that they're not going to be hit with a huge bill. That's a reasonable position for a business to take, and they're completely free to do it, but you can't also argue that customers are wrong to avoid deploying to AWS because they're scared of a surprise bill.
A simple example: S3. Currently, what happens when you exceed your credit card limit is that they send you an email they weren't able to charge you but they continue to provide the service for the next two months, and during that time everything works fine: you can access your buckets and you are charged for storage and transfer.
Now, what would happen with hard caps implemented? You didn't pay, so you're locked out of your account. Nobody can access your S3 objects, including yourself. If you care enough about them, you need to make a payment and settle your account. If you don't do it within one month, the whole content will be deleted.
Here's how to implement it:
"Amazon, what do you do today if my credit card fails and all the retries fail?"
Do THAT if billing hits <my emergency off switch threshold>.
Will it disrupt the heck out of all my AWS services? Of course. That's the point, if something went so seriously wrong that my billing hits an absurd level that will put me out of business, I'd rather have downtime.
2.5 years later, after dozens of automated mails, they finally suspended it.
If you want a cut-me-off set to $X, and some lag might allow charges to reach X+Y before the cutoff took effect, which is the customer-centric answer:
a) don't offer ANY cap, simply let the customer's out of control charges just keep racking up to catastrophic levels that put them out of business?
b) cut it off as soon as you DO detect it exceeded their cut-me-off threshold even if by that point it has reached X+Y?
Cloud vendors obviously don’t want this.
They're usually quite happy to increate the quote if you contact support.
Service limit increases are typically only denied when raising the limit would negatively impact the availability of the service (noisy neighbor issues, for example), or if the customer is needing a limit increase because they're trying to use the service in a way it wasn't designed.
and it takes GCP a day to report billing to consumer. they can monitor the data earlier than that, and stop the services early
i looked into this a year ago and there was no such thing available (there is https://docs.microsoft.com/en-us/azure/cost-management-billi... but that is not really a spending limit, it's more like in some cases you get credits from microsoft and when you have spent all the credits they stop things for you). i mean a system where you pay monthly what you consumed, and you can set a limit, and the provider guarantees that you do not have to pay more than the limit.
but maybe i overlooked something, so if you know more about this, please tell.
btw. fly.io has a kind-of spending-limit, you can preload some credit into the account and when that is spent you are relatively safe ( https://community.fly.io/t/can-i-set-a-billing-limit-per-mon... )
I would not use AWS for private stuff because of this, it is simply not worth it to risk huge bills to me personally. Of course what I might spend on AWS is peanuts, but it could translate into using AWS with my employer because I'm familiar with certain parts of it.
And even professionally this is kind of a risk, especially if you're working for a small company. And even more if you're not that experienced with AWS, there's a lot of parts to it and it can be really hard to figure out what is costing you money exactly, if you're not an AWS expert. I once created a relatively small, but noticeable charge on AWS due to a recurring script transferring a lot more data than intended. With my limited knowledge it was pretty much impossible to figure out the exact source from AWS tools. I more or less stumbled upon the issue randomly while looking into this, which is kinda scary to me.
All this does make AWS less attractive compared to classic hosting/renting a server, if that fits your use case and the lesser flexibility isn't an issue.
I struggle to consider a scenario where setting up a limited liability organisation wouldn't make sense, even if only for interacting with Amazon.
More often than not, I hear stories about how AWS support zeroed out the bill instead.
As sibling reply also says, even if they do eventually zero it out, do you want to owe the richest guy on the planet $100k? He has law firms like you have pairs of socks.
Obviously everything I'm saying goes for unlimited-billing cloud infrastructure, where your customers (or your mistakes) generate cost out of view.
[1]: https://www.ukpostbox.com/address/business-address-service
Think about the details... if you are paying something for both storage and bandwidth, and get a sudden surge of bandwidth, do you really want items in storage to be deleted? You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount; limits on its maximum size and alerts are much better.
But once you realize that bill capping doesn't make sense for storage, well, many different services are essentially some type of storage. This is a feature that sounds good but in practice what people really need is something slightly different.
I'd think you want something like a circuit breaker.. steady costs or slowly increasing costs aren't like to be a problem, but a sudden surge in costs is what I'm concerned with.
Perhaps if it operated like a time-averaged quota.. don't let me incur _new_ costs if I slide too far out of my apparent range. Give me a knob to control that range or temporarily disable it, and maybe a way to monitor those events so I can react to them appropriately for my particular application.
Actually deleting things should be done after a period of time specified in writing in the contract with the customer, and I'd hope you'd try to contact them first.
But I think an automatic bill cap absolutely makes sense for storage, and I can't really see any reasons it couldn't work.
You’re assuming the existence of a function which can extrapolate the daily/monthly/etc costs.
If I upload a 10GB file at $0.10/gbmo, how much will it cost me? We have no idea because we don’t know how long it will be up there.
We have a process that downloads and streams massive reports into S3, then a process is immediately kicked off to handle processing and importing. As soon as that completes, the files are deleted.
So we’re paying about 0.04 months of storage every month. The naive solution is estimating 1.00. You’re off by a factor of ~24.
We could... not handle that. Now we’re in a situation where we’re consistently spending $100/mo on storage, set a liberal $150 budget and immediately everything breaks. We actually needed to set a $2400 budget! Expecting a $100 bill and getting a $2400 bill some month when something goes wrong is as good as doing nothing.
You’re also going to need to account for the interaction of every single object in a bucket and the lifecycle rules set which is an absolutely ridiculous amount of overhead.
Now extend this to... literally every one of the seemingly bazillion AWS services.
The only cap that’s going to be easy and realistic to implement without a crystal ball is a hard cut-off once you’ve actually accrued the charges, and a “shut down all my servers, delete all my data, and just close my account” cap is useful to almost no one.
If you want useful caps they need to be aware of your workload. The best person to implement those is you. The tools are there.
If I estimate I'm going to need to spend $400 a month on my bill, then I might set a limit of $800, send me an email warning when I'm getting close, I'll go in and work out why I'm consuming much more than I expected. If I don't hit the "let me exceed my billing limit this month" button then stop the service. (but don't delete any data, I'm not sure why so many commenters seem to think this is a necessary step).
Amazon is capable of calculating how much to charge you every month, so they're capable of doing this too. No fancy estimation required.
==================================
So under your example - I upload my 10gb file, but some bug in my processing code means it doesn't get deleted and sits there for a while. Maybe I don't notice and it sits there for a few days instead of a few seconds, but Amazon sends me an automatic email because my usage this month so far is X% higher than the average of my last Y months.
Ideal scenario: I go in, investigate, delete the file and temporarily increase my billing cap just for this month.
Not great, but okay scenario: I don't notice the email, a few days later and I'm at ~2x my previous months bill (or whatever limit I've set up), amazon automatically starts returning 402s when I try to access my storage service. I am happy because this is still much better than getting a bill for 10x my previous amount at the end of the month instead.
You're accruing costs even if you stop reading and writing. You're paying for the on-going storage, running of instances, etc.
What you're describing does not implement what you're describing it as.
E.g.,
It's the first day of the billing cycle, so our bill is effectively $0. So no limits in place on writing/rates/etc. We upload 100TB of data. No rate limiting or blocking because our current bill is under the $800 limit.
By about the 8th of the month, we've now hit the $800 limit you set. If you don't go hit the "let me exceed my billing limit" then you want Amazon to "stop the service".
You want:
1. Under no circumstances to spend more than $800/mo.
2. Your data to be retained.
* Amazon retains your data: You need to continue paying for storage. Your bill at the end of the month will be ~$2200. Fails #1.
* Amazon "stops the service" (storing your data) and under no circumstances exceeds $800/mo in charges: They delete your data. Fails #2.
Alternatively, if you want them to not accept a write if it _would_ have led to you being over your budget, see my original comment about predicting the future.
Billing alerts take two seconds to set up in CloudWatch.
If it's a legitimate goof, contact support and they've refunded me every time.
Glad we're all on the same page then and there's no problem here.
Also interested to hear exactly what part of what I said was "twisting" or misrepresenting the issue.
They provide billing estimates and billing data through CloudWatch metrics. This can be used to alert via SMS, email, and other methods. These can be used to trigger lambda functions to implement your own "shut it down" functionality quite easily in a way that actually makes sense for your workload.
What you seem to be advocating for is, more or less, a "pay what you want" model. If they're going to provide services and let you choose the maximum you're willing to pay and expect them to eat the rest, then I don't know how else to describe it.
In most use cases you're right. But never? Not true, not all storage is used the same way. And the thing is, why should Amazon decide which use case is valid or not? I might care more about being able to afford the service than to keep the data around, depending on what kind of service I offer my user. As a platform provider, they should be agnostic, but their greed for money is shining through their willingness to be a true platform provider.
But couldn't Google/Amazon simply pretend they have real-time bill capping, and then simply swallow the costs incurred by any delays?
Doesn't seem like rocket science, probably won't cost much, and might bring in new kinds of business (a lot of businesses won't allow most employees to sign an effectively blank check)
Now that I think about it, a third party could do this as well... except that if it were built in to AWS there would be much less friction.
So you hit your limit. Is all your storage deleted? You can't receive an alert because that costs something (even if it's a fraction of a cent) to send. Are your domains forfeit? Audit logs destroyed? There's no reasonable way to implement this. Billing alerts are the best you can do on this problem and I think it would be a good faith move for AWS to enable some by default but a limit just doesn't make sense on any level.
EDIT: Lots of people proposing solutions that work for them. AWS has to think of everyone. And they did. That's why budget alerts exist and you can respond in the way you choose. Everyone's conflicting ideas for how to solve this can be implemented today on top of billing alerts/actions[1]. Case closed.
[1] https://aws.amazon.com/blogs/aws-cost-management/get-started...
> So you hit your limit. Is all your storage deleted?
Easy. Cut access, don't delete the files, set a time limit for resolving the problem before they are deleted.
> EDIT: Lots of people proposing solutions that work for them. AWS has to think of everyone.
No they don't. A feature that solves the main problem for lots of users can be added and for the users that need a complex solution, well, they can justnot use the simple one.
1. Misconfigured software that we accidentally pushed into production and by the time we noticed it like an hour later we were $100s out. Luckily it was not in the 1000s other that would have sunk the business. 2. An old account of ours got hacked and people spun up like 50-100 servers and it happened overnight and we had a bill of $50k-$70k. Luckily Amazon wrote that off for us. Otherwise I have no clue how we would have ever paid that.
These are usually extraordinary cases, so even if there is a data loss or your servers are shut down, it is a better option than going down the hole with 10s of 1000s in charges.
Of course, it's no more realistic to charge people for the alert than it is to charge them for email receipts. Do you not get those from Amazon?
You implement the feature cap by ceasing to provide the feature once the account is over the billing limit. It is always possible to project the cost of "freeze everything until the end of the month", because there is no unknown information in that scenario. Nothing you've listed is an actual problem.
Yea, what a crazy world that would be if AWS just gave each account free stuff like free data transfer for the first gb, or free s3 storage for the first ~100mb
A limit very much makes sense of many levels, just not all of them at once. You're arguing against a hard limit of every single AWS service.
Nobody wants their storage deleted once their budget hits a limit, what they want, and is reasonable to have, is a limit that prevents AWS from spinning additional EC instances. Or a limit that blocks your usage of Textractor. Or many other limits an user could individually set-up for the services where a limit makes sense.
> AWS Budgets now allows you to configure actions — responses to cost and usage in your account or set of accounts— that will be applied automatically or via a workflow approval process once a budget target has been exceeded. There are three action types: Identity and Access Management (IAM) policies, Service Control policies (SCPs), or target running instances (EC2 or RDS). Actions can be configured for actual (after they’ve occurred) or for forecasted (before they occur) budgeted amounts.
https://aws.amazon.com/blogs/aws-cost-management/get-started...
Anyway, they should be able to protect people from an unexpected $1k charge. Once you deal with alerts triggering automatic infrastructure changes, $1k is likely a value you won't even notice.
There is no reason why an absolute "don't ruin me financially" button can't be implemented in addition to everything you mention.
>There's no reasonable way to implement this.
Azure VS enterprise credits work this way. 150 hard cap with no credit card on the individual account.
The problem is that you can run into unexpectedly high cost without warning or option to decide whether it is worth it and one of the solutions would be to just tell people when the rate at which they use your stuff increases or decreases by a certain threshold. And because they can bill you, they also know (roughly) what the things that you use cost.
I never used AWS for serious things, but not being able to decide my spending will be a serious argument against it.
> There's no reasonable way to implement this [...] on any level
That's not true (as OP proposes a reasonable way)
This also aligns well with the "planned vs unexpected" costs.
Sure. That's why you make it optional. And configurable in a way
But at a very basic idea, blocking new services from being started and throttling of existing ones would go a long way in helping. A very long way.
None have the engineering capacity to figure out how to cap the spending on each one of the individual services, each with their unique and special API.
You're arguing that Amazon can't afford the difficult engineering of spending caps, but the very customers that need spending caps because their budgets are so constrained are so well moneyed that they should all individually be able to invent a solution, engineer it, and manage it?
Oh, really? Azure wants to have a word with you ... Yes, currently they don't enable this for subscriptions with commitment plans or with pay-as-you-go pricing, but it is not because it is not possible or feasible, as you argue - the technical infrastructure in the form of spending limits, spending budgets and quotas is there [1-3] and is available for select plans [4].
[1] https://docs.microsoft.com/en-us/azure/cost-management-billi...
[2] https://docs.microsoft.com/en-us/partner-center/set-an-azure...
[3] https://docs.microsoft.com/en-us/azure/azure-resource-manage...
[4] https://azure.microsoft.com/en-us/support/legal/offer-detail...
But yes, you could set up alerts to monitor when this is likely to occur. But to do this across each service for each reason your bill might spike drastically might be a challenge.
In the linked article the user was alerted, but they were asleep. One of the proposed solutions is to have usage limits which are calculated based on a maximum monthly cost and AWS will work itself within that.
The worst case scenario is that the user configures their limits in a poor way then turns around and blames Amazon when something breaks in their project and Amazon points back at the limit. It would just be bad all around...so all in all, I think alerts are the best way to do this...
And where are your code reviews?
If that's the case then why hasn't it been responded to? Is responding also nonsensical?
At one of the companies years ago a developer accidentally leaked an API credential on GitHub and the company woke up with a $30k bill.
Eg just have 2 caps: First cap triggers a red alarm prohibiting provisioning, or storing, of new data. The second one (when you are like 50% above the first cap) is dark red, closing down everything except storage.
Sure, decisions have to be made for every single service, but you don’t need a perfect solution if it is optional.
Eg, you could start ONLY with EC2 and will have fixed 50% of the problem.
Currently it's too much of a hassle to increase/decrease account-wide limits: it takes too much time, depends on AWS Support, UX is nowhere as nice as, say instance reservations etc.
As a result of this people tend to increase the limits once in a while just to make sure it doesn't become an issue at a very unfortunate time, and that's it. It never gets decreased/managed again. It's obviously not seen as a use-case by AWS, since they only have a "Request quota increase" button in the UI, and no "decrease" button. [1]
SQ on its own wouldn't cover all kinds of costs (for on-demand items like Lambda executions they offer limits on rates, not total count), but it'd be better than nothing, it'd prevent scenarios where you'd wake up to 100 GPU instances mining bitcoin, and it's within the quotas since Bob asked for an increase two years ago to try something.
[1] Just having a button there would still be an awful UX, yet I believe its complete omission is noteworthy.
Borrow what DHCP does for leases, but with payment quotas.
First layer is a burst quota, up to that maximum in any spike. This only regenerates when the (monthly?) average quota has enough slack at current that an additional burst can be harvested without the remaining average being under the initial set value. (The math would probably be different, remaining period quota less burst average over time greater than period quota average, but that's the sales description of the concept.)
The second layer is the monthly (or some other period) average for the maximum expenditure allowed.
A billing endpoint would maintain a fractional bucket of spending (divided up as makes sense) but in the case of a single quota consumer would receive an estimate for the period (ideally the burst) and allow up to half of that to be used. At that point it will 'renew' the quota (lease), and flush billing, including sending alerts. If there isn't any further quota the remaining released balance would be consumed and then requests fail.
that's but a drop in the digital ocean
I think it’s tricky to implement, though.
When it reach the cap, all your service stop working? This is incredibly user hostile.
If you want some warning when utilization is high, it can be relatively easy to setup with AWS's existing feature, but you need to do some configuration/tweaking on your own.
Not nearly as user hostile as the possibility of an unexpected DOS on your bank account while you're asleep.
Which has actually happened to people.
My expertise is not at all anything related to web development so when I first tried experimenting with AWS for a side project I was terrified. It was an experiment for me and i didn't care if it deleted the data since the data it was storing was test data I was uploading. However, my budget for my entire side project was $500 and it seemed totally feasible (at least to a first time user of such a service) that some wrong lines in the code could balloon the requests or storage I made. At the time, I didn't even have $5000 of liquid assets and it would have been a nightmare.
So set your cap to a very high number, much higher than any amount you expect to be billed. Surely for everyone there is some number at which they would rather their services go down, than have to pay it?
That exactly what I would like to have. Services stop and I get time to review what's happened without any stress that my bank account will be emptied.
Having a killer switch that could make one's AWS account stop would be a risk for big customers, I would predict, if it is misconfigured, maybe even intentionally. A better solution for them would be alerts, and get things right afterwards.
The AWS support was super nice and took care of it for me but it gave me some sleepless nights until it was resolved.
If this was a feature I would have most definitely capped my account at 1k (my max bill being $50). They can always send alerts to increase your cap based on projections like it shows in cost explorer. Also since it's an opt-in feature I'm kinda taking responsibility that I'm okay if services are suspended due to hitting the caps.
The lack of caps is just laziness or a dark pattern by Amazon. There’s no excuse to not implement them, it’s not a hard problem.
Also, I can imagine clients worried about this are not the money-making kind of clients anyway.
Another issue is how often does the quota reset? Daily? Hourly? Monthly? That’ll determine how big those spikes are when the quota lifts.
I've worked on account leasing systems for temporary accounts and talked to the internal team for AWS event engine. And they don't have much more than the official tools.
Edit: not been..
Well, that's what the incentives would suggest.
* Don't give companies that do this money
One of the biggest questions here is what are customers asking for? And the necessary follow-up question: Do customers want to actually bear responsibility for their choices?
Recently, I had a customer lodge a support case because their programmatic access keys were leaked. A malicious actor was then able to use those credentials to exfiltrate their S3 data, and delete it. Now the S3 data is being ransomed.
The customer opened a case asking if there was any way to get the data back. If we had to go the 'extra mile', the customer demanded we do that.
The answer is simply: No. We can't get that data back.
Customers demand that they own the data they upload into S3. They don't want AWS to be able to read the data, nor do they want us to store the data internally as a backup. That's what customers demand, and what AWS gives them.
Now something regrettable happened (a customer got pwned) and now a customer wants AWS to bear the responsibility for backing up the data. I bet a week ago they would've demanded that they have absolute sovereignty over their data. Shit changes when shit hits the fan.
---
Hard caps are doable, but the question is: Do customers want responsibility for this feature?
I read a comment wherein someone's startup was killed due to $30k of bandwidth costs because of misconfigurations that led to users abusing their platform. That sucks. It's not in AWS's interest to put their customers out of business.
But let's look at the flip side. What if someone in the finance department puts a hard cap on an account. But then a tweet goes viral, and business is pouring in. Everyone, in their rush to keep everything scaling appropriately, forgets there's a hard cap. They hit the business and now there's a hard outage in the middle of the biggest business event they've ever had.
Who's the customer going to hold responsible for the outage? As someone who deals with customer misconfigurations on AWS for a living, I assure you the customer won't be calling themselves demanding an explanation.
What happens if the person in charge of budgets doesn't lift the cap before the Christmas holidays? Again, hard down. Again, customer won't be mad at themselves.
---
It's better to let AWS run as the customer configures it to run than bring everything to an abrupt stop. Billing alerts exist, and you can use Lambda to turn off resources that are above their billing threshold.
But doing a hard cap on an account? Something a subsection of customers might want, but something not many customers will want to take responsibility for when something goes sideways.
Well, I think the same point can be made here: if you need a price cap, then AWS is too expensive for you ;)
The problem is that AWS markets itself as “pay what you use” and generally that’s extremely little for me, but if I screw up with Glacier I suppose I could get a $100 bill because I restored some backups the wrong way.
You wouldn't be the first person to do that: https://medium.com/@karppinen/how-i-ended-up-paying-150-for-...
The whole idea of services like AWS is that it is scalable as a solution for startups, small businesses and large global corporations. Not just large corporations that would be considered customers of a "fancy" web services provider.