On the other hand, it sounds like you hired someone who wasn't really up for the level of responsibility given. :(
In theory ;), you shouldn't have to be a core expert in everything. But yeah... in the real world, things aren't so cut and dry. :/
In theory ;), you shouldn't have to be a core expert in everything. But yeah... in the real world, things aren't so cut and dry. :/
Easier for both sides to just ask AWS for a refund if there's a reasonable case.
This wouldn't be an issue if it was configurable.
Would you rather make a mistake leading to a big bill with the possibility of a refund or set your max budget and have your resources permanently deleted?
EC2 instances, EBS volumes, S3 data... should AWS delete those when you hit your budget? How do you stop the billing otherwise?
With prioritisation, so the non-steady state services are stopped/killed with plenty of time to leave the needed foundations still running. :)
2) If it's a soft budget then it's no different than the alarms you already have.
3) If you want to stop it before it hits the budget, then you're asking for a forecasted model with a non-deterministic point in time where things will be shutdown.
This just leads to neverending complexity and AWS doesn't want this liability. That's why they provide billing alarms and APIs so you can control what you spend.
Not if I'm busy, or away from work, or asleep. There is a massive difference between getting an alarm (which is probably delayed because AWS is so bad at reporting spent money) versus having low priority servers immediately cut.
Even without a priority system, shutting down all active servers would be a huge improvement over just a warning in many situations.
You want it to selectively turn off only EC2? Does it matter which instance and in which order? What if you're not running EC2 and it's other services? Is there a global priority list of all AWS services? Is it ranked by what's costing you the most? Do you want to maintain your own priority of services?
And what if the budget was a mistake and now you lost customers because your service went down? Do you still blame AWS for that? Or would you rather have the extra bill?
There is no easy solution.
"Everything except for persistent storage" is nowhere near useful enough to work and can cause catastrophic losses. Wipe local disks? What about bandwidth? Shutdown Cloudfront and Lambda? What about queues and SNS topics? What about costs that are inseparable from storage like Kinesis, Redshift, and RDS? Delete all those too? And as I said before, what happens if you set a budget and AWS takes your service down which affects your customers?
It's easy to say it's simple in an HN comment. It's entirely different when you need to implement it at massive scale and that's before even talking about legal and accounting issues. There's a reason why AWS doesn't offer it.
I sometimes for example fiddle with Google APIs. I do not even have customers so don't really care if things will stop working, but I have accidentally spent 100 euros or more. I have alerts, but those alerts arrived way too late.
I make a loop mistake in my code and now I suddenly owe 100 euros...
I literally just explained why this doesn't work with AWS services. You will have data loss.
And it creates a whole new class of mistakes. If people mistakenly overspend then they'll mistakenly delete their resources too. All these complaints that AWS should cover their billing will then be multiplied by complaints that AWS should recover their infrastructure. No cloud vendor wants that liability.
ADDED: A lot of people seem to think it's a simple matter of a spending limit. Which implies that a cloud provider can easily decide:
1.) How badly you care about not exceeding a spending threshold at all
2.) How much you care about persistent storage and services directly related to persistent storage
3.) What is reasonable from a user's perspective to simply shutdown on short notice
Except it wouldn't. This is the 3rd time in this thread explaining that. Edge cases matter, especially when creating leading to new mistakes like setting a budget and deleting data or shutting off service when customers need it most.
If it's not a hard budget but a complex set of rules to disable services... then you already have that today. Use the alarms and APIs to turn off what you don't need.
> If it's not a hard budget but a complex set of rules to disable services... then you already have that today. Use the alarms and APIs to turn off what you don't need.
I have been describing a simple set of rules, not a complex one.
It used to be extremely difficult to get accurate usage data on all their services. Has that been fixed? If not, then the alarms aren't good enough. If the alarms can automate enough right now, in a non-buggy way, then that should be the answer to people "hey, the alarms do more than alarm, use them to trigger shutdowns". Don't say "it can't be done, sorry". If the alarms aren't good enough for that automation, then the argument stands.
And using the APIs means that each company that wants safety is duplicating effort in an almost untested way, a recipe for so many bugs it makes the problem worse. No, this needs to be a feature of AWS itself.
Not so simple.
That's not a he's cap, since turning off services isn't instant and costs continue to accrue. But, yes, there are ways to mitigate the risk of uncapped costs and they are subject to automation.
AWS would rather lose some billings than deal with the fallout of losing data or critical service for customers (and in turn their customers).
In theory I could build something using budget alarms, apis, and iam permissions to make sure everything gets shut down if a developer exceeds their budget, but if I made a mistake it could end up being very expensive. Not that I don't trust developers at my company to use such an account responsibly, but it is very easy to accidentally spend a lot of many on AWS, especially if you aren't an expert in it.
It's impossible for AWS to know how to handle hard caps because there are too many ways to alter what's running and it's too contextual to your business at that moment. That's why they give you tools and calculators and pricing tables so that it's your responsibility (or a potential startup opportunity).
Money is easy to deal with. Alarms work. Bills can be negotiated. But you can't get back lost data, lost service, or lost customers.
Right. In my experience, if you don't understand what's going on beneath your abstractions, you're always in for a world of hurt as soon as something goes sideways.