Hacker News Reader

Comment by nanidin | Hacker News Reader

It has been a while since I messed with the AWS panel, but IIRC you can set budget alerts so you’re notified once a threshold is crossed. It’s not a perfect solution, but if you expect to spend $0 then an alert on $0.01 is pretty trivial to set up and goes a long way to prevent end of the month surprises.

View on HN

Yes, you can set up a budget alert. I have it set up for one of my work accounts. Problem with that is that it's an email, and there's usually some delay in it going out. It's not a great solution for accidental traffic spikes (surprise HN post, or someone finding and abusing a public S3 object you have hosted, etc).

So you get an email saying your $10/mo site is now $1000 for this month, and climbing.

In general I wouldn't recommend using AWS and expecting the free tier for anything that's going to be public facing or autoscaling.

It’s not a perfect solution

It's not a solution at all given the fact the alerting process can lag behind the logging process by several hours or more. If you've hit a traffic spike it could have rolled over your site and gone in that time, leaving you with a bug bill.

Alerts are not a viable solution to traffic spikes unless they're real-time and absolutely bulletproof. AWS's alerting is neither.

An alert is not a cap. They should add a cap to just shut down/delete resources if the cap is hit.

You can use Budgets Actions for that:

https://aws.amazon.com/blogs/aws-cloud-financial-management/...

Note that these are not instantaneous. You can still incur charges before the budget action kicks in and terminates your resources.

True. But you can do it on forecasted costs.

Ok so how would this work in any other way?

Should _every_ S3 action have a `if will_incur_charges() and should_not_incur_charges(): raise Exception()` statement in it's critical path? No, of course not. Everyone get's slower for nobodies benefit. It has to be delayed.

But then you run into an issue: what if you end up costing AWS 100$ before the budget action kicks in. Should you not pay that? Why not?

It’s a business decision on AWS’s part. If they want to reduce cognitive load or objection of some customers, they’ll be more willing to “eat it”.

My strong guess is if you had a free account, setup a budget cap, went over it, and they decided to charge you*, a quick email to support would get it waived.

I’m very much a fan of AWS, in part because while they have the chance to uphold the legal terms, my experience is that they’re pretty customer friendly.

* Early on, I had many bills under $1/mo that they just comped without me having to do anything.

> Should _every_ S3 action have a `if will_incur_charges() and should_not_incur_charges(): raise Exception()` statement in it's critical path?

We're talking about one account-wide flag `has_exceeded_billing_limits`. Changes are infrequent, and can be pushed into caches. Small overruns while the flag pushes are trivially eaten by AWS.

You claim that a bunch of GPU crypto minors won't exploit a chance to incur billing overruns if amazon waives charges?

The amazon deal is simple. Very clear pricing for pay what you use.

Cloudflare - can you link to the page where they show the cost of bandwidth? Still waiting.

> Should _every_ S3 action have a `if will_incur_charges() and should_not_incur_charges(): raise Exception()` statement in it's critical path?

Budget actions work by applying an Deny All to IAM, which has essentially exactly that.

The problem is not the shutting down, it's the detection. AWS billing has a resolution measured in hours, which has limited usefulness on a platform where you can rack up thousands of dollars in charges in just a few minutes.

And if you get in a car accident and don't check your email for a few days? It's extremely hard to understand how a company with their resources is able to send a notification but not able to shut it off if that's what the customer wants.

Its not hard to understand at all, they willingly screw over customers, just look at all the posters here falling all over themselves to excuse this blatantly shitty customer hostile behavior.

Well they're two different problems. For AWS to give you an alert a service X needs to send billing data to the billing system.

For the billing system to then "turn off" X it needs a number of things.

1. It needs the ability to reach back out to that service. It probably has no idea what it is, all the billing system is likely to receive is something like:

    {service_name: "X", action: "Put"}

ie: Pretty opaque data with just enough structure to know "This costs X cents and happened Y times".

So now your billing system needs to be able to resolve "X" back to some AWS resource that it can talk to. Both the resolution of X as well as the "billing can now talk to every single AWS service" are pretty heavy lifts.

2. It needs to know what "off" is. "Off" for S3 could mean a lot of things.

a) Delete the bucket and all data inside of it

b) Keep the bucket but delete all data inside of it

c) Keep the bucket and the data but disable API access

etc etc. Do you disable PUT? GET? Both? What if what's blowing up your billing is GET?

And this really doesn't get easier for other systems. Do you back up a database before killing it? That incurs charges too.

I don't see AWS somehow solving this in a "one size fits all" way because there isn't one.

Admittedly, all of these are real issues. The thing is that for a company with 1.816 trillion (1816 billion!) market cap as of today, all of these issues are easily solvable. But it's not a matter of "we can't solve it" or even "we don't have the resources to do it, there is some higher priority problem to tackle first".

It's not an engineering problem at all at its heart.

It's a marketing/business problem that someone somewhere is thinking that Amazon can provide a free service to X users, knowing that Y (X, Y positive, Y << X) users will go over their "free tier" usage and pay for all X's costs, maybe even more, making the free tier a profitable business on its own.

> It's not an engineering problem at all at its heart.

My point is that it is definitely an engineering problem as well as a product problem.

a) It's going to be super technically difficult to build (especially in a way where it's responsive at a granular basis to handle huge blow-up bursts)

b) It's not even clear what you're supposed to be building

None of what people are proposing is well defined or easy to build.

The type of questions you are mentioning are basically saying different people would need different configuration options and come up every time you design any complex technical feature that is going to be used by millions of people. These are mostly product and UX questions, and NOT engineering ones. The only engineering problem I can think of is the latency in getting real time billing information.

The way you usually solve this is by having sane defaults, and giving users different mechanisms for configuration based on how complex their configuration needs are. This can take a tiered approach.

As an example, simpler and straight forward things (such as disable egress traffic from S3 if the bill exceeds X) can be in the UI itself. Then, for customers who need more control, an option to configure via json or yaml similar to cloud formation. For anyone who needs even more giving an option to call a customer defined lambda function would give them the ability to at any metrics and take appropriate action.

> NOT engineering ones

Engineering problems:

1. How do you actually have a billing system reach out to every other system? It has to resolve the resource, network with it, have IAM permissions, a network route, etc.

2. How do you handle consistency?

3. How do you make it responsive?

4. How do you add this to every billable entity?

I mean, it's just a shitload of work, and all of that just to get to a terrible idea.

> The way you usually solve this is by having sane defaults, and giving users different mechanisms for configuration based on how complex their configuration needs are. This can take a tiered approach.

The sane default is you pay for what you use, and you can listen to billing events and build all of the logic you're talking about if you want to.

Reply on news.ycombinator.com