Hacker News Reader

Comment by Epskampie | Hacker News Reader

AWS free tier is horrible. You have to enter a creditcard to sign up, and then there is no way to prevent it being charged when you go over the limit for some reason. If you get on the frontpage of HN for example, you might be majorly screwed.

It's a test account, I just want it to shut down when the limit is reached.

View on HN

I think AWS drastically needs to create some type of "sandbox account" flag that severely locks down the services you can use and the amount you can scale up, exactly for reasons like you said.

However, I also think a big problem is that many people on the internet and especially people who try to sell AWS tutorials or learning courses push AWS as some toy that every developer should sign up for on a whim without understanding what they are doing. An AWS account is an industrial-grade tool, it's not a toy, and it should be treated as such. It's like renting a backhoe when you don't even know how to use a shovel yet, and then being surprised when you completely screw up your yard.

Sites like acloudguru that offer ephemeral sandbox AWS accounts are becoming more popular, and people new to AWS should really be steered towards those.

It has been a while since I messed with the AWS panel, but IIRC you can set budget alerts so you’re notified once a threshold is crossed. It’s not a perfect solution, but if you expect to spend $0 then an alert on $0.01 is pretty trivial to set up and goes a long way to prevent end of the month surprises.

An alert is not a cap. They should add a cap to just shut down/delete resources if the cap is hit.

You can use Budgets Actions for that:

https://aws.amazon.com/blogs/aws-cloud-financial-management/...

Note that these are not instantaneous. You can still incur charges before the budget action kicks in and terminates your resources.

True. But you can do it on forecasted costs.

Ok so how would this work in any other way?

Should _every_ S3 action have a `if will_incur_charges() and should_not_incur_charges(): raise Exception()` statement in it's critical path? No, of course not. Everyone get's slower for nobodies benefit. It has to be delayed.

But then you run into an issue: what if you end up costing AWS 100$ before the budget action kicks in. Should you not pay that? Why not?

It’s a business decision on AWS’s part. If they want to reduce cognitive load or objection of some customers, they’ll be more willing to “eat it”.

My strong guess is if you had a free account, setup a budget cap, went over it, and they decided to charge you*, a quick email to support would get it waived.

I’m very much a fan of AWS, in part because while they have the chance to uphold the legal terms, my experience is that they’re pretty customer friendly.

* Early on, I had many bills under $1/mo that they just comped without me having to do anything.

> Should _every_ S3 action have a `if will_incur_charges() and should_not_incur_charges(): raise Exception()` statement in it's critical path?

We're talking about one account-wide flag `has_exceeded_billing_limits`. Changes are infrequent, and can be pushed into caches. Small overruns while the flag pushes are trivially eaten by AWS.

You claim that a bunch of GPU crypto minors won't exploit a chance to incur billing overruns if amazon waives charges?

The amazon deal is simple. Very clear pricing for pay what you use.

Cloudflare - can you link to the page where they show the cost of bandwidth? Still waiting.

> Should _every_ S3 action have a `if will_incur_charges() and should_not_incur_charges(): raise Exception()` statement in it's critical path?

Budget actions work by applying an Deny All to IAM, which has essentially exactly that.

The problem is not the shutting down, it's the detection. AWS billing has a resolution measured in hours, which has limited usefulness on a platform where you can rack up thousands of dollars in charges in just a few minutes.

Yes, you can set up a budget alert. I have it set up for one of my work accounts. Problem with that is that it's an email, and there's usually some delay in it going out. It's not a great solution for accidental traffic spikes (surprise HN post, or someone finding and abusing a public S3 object you have hosted, etc).

So you get an email saying your $10/mo site is now $1000 for this month, and climbing.

In general I wouldn't recommend using AWS and expecting the free tier for anything that's going to be public facing or autoscaling.

It’s not a perfect solution

It's not a solution at all given the fact the alerting process can lag behind the logging process by several hours or more. If you've hit a traffic spike it could have rolled over your site and gone in that time, leaving you with a bug bill.

Alerts are not a viable solution to traffic spikes unless they're real-time and absolutely bulletproof. AWS's alerting is neither.

And if you get in a car accident and don't check your email for a few days? It's extremely hard to understand how a company with their resources is able to send a notification but not able to shut it off if that's what the customer wants.

Its not hard to understand at all, they willingly screw over customers, just look at all the posters here falling all over themselves to excuse this blatantly shitty customer hostile behavior.

Well they're two different problems. For AWS to give you an alert a service X needs to send billing data to the billing system.

For the billing system to then "turn off" X it needs a number of things.

1. It needs the ability to reach back out to that service. It probably has no idea what it is, all the billing system is likely to receive is something like:

    {service_name: "X", action: "Put"}

ie: Pretty opaque data with just enough structure to know "This costs X cents and happened Y times".

So now your billing system needs to be able to resolve "X" back to some AWS resource that it can talk to. Both the resolution of X as well as the "billing can now talk to every single AWS service" are pretty heavy lifts.

2. It needs to know what "off" is. "Off" for S3 could mean a lot of things.

a) Delete the bucket and all data inside of it

b) Keep the bucket but delete all data inside of it

c) Keep the bucket and the data but disable API access

etc etc. Do you disable PUT? GET? Both? What if what's blowing up your billing is GET?

And this really doesn't get easier for other systems. Do you back up a database before killing it? That incurs charges too.

I don't see AWS somehow solving this in a "one size fits all" way because there isn't one.

Admittedly, all of these are real issues. The thing is that for a company with 1.816 trillion (1816 billion!) market cap as of today, all of these issues are easily solvable. But it's not a matter of "we can't solve it" or even "we don't have the resources to do it, there is some higher priority problem to tackle first".

It's not an engineering problem at all at its heart.

It's a marketing/business problem that someone somewhere is thinking that Amazon can provide a free service to X users, knowing that Y (X, Y positive, Y << X) users will go over their "free tier" usage and pay for all X's costs, maybe even more, making the free tier a profitable business on its own.

> It's not an engineering problem at all at its heart.

My point is that it is definitely an engineering problem as well as a product problem.

a) It's going to be super technically difficult to build (especially in a way where it's responsive at a granular basis to handle huge blow-up bursts)

b) It's not even clear what you're supposed to be building

None of what people are proposing is well defined or easy to build.

The type of questions you are mentioning are basically saying different people would need different configuration options and come up every time you design any complex technical feature that is going to be used by millions of people. These are mostly product and UX questions, and NOT engineering ones. The only engineering problem I can think of is the latency in getting real time billing information.

The way you usually solve this is by having sane defaults, and giving users different mechanisms for configuration based on how complex their configuration needs are. This can take a tiered approach.

As an example, simpler and straight forward things (such as disable egress traffic from S3 if the bill exceeds X) can be in the UI itself. Then, for customers who need more control, an option to configure via json or yaml similar to cloud formation. For anyone who needs even more giving an option to call a customer defined lambda function would give them the ability to at any metrics and take appropriate action.

> NOT engineering ones

Engineering problems:

1. How do you actually have a billing system reach out to every other system? It has to resolve the resource, network with it, have IAM permissions, a network route, etc.

2. How do you handle consistency?

3. How do you make it responsive?

4. How do you add this to every billable entity?

I mean, it's just a shitload of work, and all of that just to get to a terrible idea.

> The way you usually solve this is by having sane defaults, and giving users different mechanisms for configuration based on how complex their configuration needs are. This can take a tiered approach.

The sane default is you pay for what you use, and you can listen to billing events and build all of the logic you're talking about if you want to.

The solution is to not use AWS. Is is not a sacrifice to not engage with shady companies.

Paying for what you use, the peak of shady business practices!

Paying for what your users use, with no way to say no to the traffic, is exactly why billion dollar companies like Cloudflare exist - to protect people from "paying for what you use".

So yes, it is shady not to protect customers from that IMO.

That's really disingenuous.

The complaint here is that Amazon offers a free tier supposedly for learning the platform, but it is a giant footgun that shoots a ton of people in the foot.

People are reasonably asking for hard limits to protect them from this highly foreseeable situation wherein a complicated cloud offering can go on a spending runaway.

It is literally as easy as following a beginner tutorial and selecting the database instance the tutorial uses and leaving it running. That could be a several hundred dollar mistake.

"Kill my service if I hit a billing limit" is a scary footgun as well, and one that could impact larger customers.

I don't think it's unreasonable to say that if you're using AWS you're taking on some responsibility to make sure you're not blowing up your bill. AFAIK you are automatically enrolled in emails that will tell you when you're about to exit a free tier limit, so it's not like they won't warn you.

> "Kill my service if I hit a billing limit" is a scary footgun as well, and one that could impact larger customers.

Then make it opt-in but a highly visible one during account creation so that people who just want to test can enable it.

Nobody is suggesting to make that the default uniformly across all AWS accounts.

It would be excellent for anybody intending to use the free tier for its stated purpose (getting to know the AWS platform) who would like to make the _choice_ to shut off their services if they are going to exceed the free tier quotas.

That way you are free to experiment, and if you blow up something while learning, you're not then leaning on the mercy of AWS support to refund you.

The free tier is not just for getting to know the AWS platform. I worked with customers hosting the entirety of their early applications on free tier services.

Playing around turns into production.

When your service goes down and you lose $XXXXXX revenue suddenly it's AWS's problem. AWS has taken the approach that keeps the lights on, assuming its customers understand their unit economics.

Right there at the top of https://aws.amazon.com/free/ is this:

  AWS Free Tier
  Gain free, hands-on experience with the AWS platform, products, and services

Just because you've worked with customers hosting in the free tier doesn't mean that all free tier users would or should choose to prioritize uptime over cost.

Make it a clearly selectable option, explaining both downsides. Be vocal about it being selected if you want to.

But the thought that not immediately reacting to an e-mail can cost me a month's salary or more is terrifying. Maybe it'll get waived. Maybe it'll be waived in the form of credit that I can only spend on the product (i.e. for my purposes, not waived). Maybe I'll be stuck with it. The "maybe" is the danger here.

Let's say I put some obscure 50 MB dataset into a public S3 bucket, and pay my <1 cent per month to host it and like a dollar for each 200 downloads. Rarely does a month exceed a dollar or two, all is good.

Then someone builds a poorly made colab that downloads the entire dataset each time it is run, and the colab hits the front page of HN while I'm traveling, and makes it to social media the next day because it shows something funny. And people don't run it just once, they play with the parameters, running it multiple times.

By the time I'm back and see the e-mail, a 10000 USD bill may be waiting for me.

To obscure? How about this one:

The operator of a semi-popular website has decided to hate me. Each page load now contains an <img src=[my image]?t=[timestamp] width=1 height=1> in the header, pointing to the biggest image I'm hosting for my small static website.

Edit: Even better example

I've accidentally left an API key in a git repo that I pushed online. My carefully set up billing alert was deleted, then my account started spun up as many of the most expensive GPU instances as quota would allow and started mining Monero. In this scenario, I think a $10000 bill would be "getting off easy".

(Just to be clear, these are hypothetical scenarios. If anyone knows how various cloud providers would react to those in practice, or if you know that there are countermeasures that would reliably stop them, please do tell!)

I think I set an alert recently on AWS emailing me if I surpass a limit that I set.

Have you checked out AWS console recently?

That's been a feature since AWS started. That isn't at all what I am talking about above.

Um. They'd much rather deal with a waiver request from someone for $100 then to deal with a bad credit card in accounting spinning down a major account.

BTW - I've never heard of the later happening at AWS ever, and I have at other hosting providers.

I don't think AWS's billing system is robust enough for that. In pure AWS fashion you have to create a billing alarm, which pushed an event to SNS which triggers a lambda to shut down your stuff but it is possible. The catch is that billing metrics are estimates and alarms might be delayed.

Yup, seems like they have a lot "eventually consistent" batch processing going on. Certain things like CloudTrail have 4-6 hour lags in billing (unless it's improved in the last year)

It'd be a huge engineering effort to make something instantaneous--I think the closest thing they have to such a system is whatever they use for rate limiting or IAM.

I'm guessing there's a pretty high overhead to trying to do realtime instead of batching

AWS oopsies suck but I think their billing system is pretty robust compared to lots of usage based billing systems (like, say, utilities)

i don't think you'll find much sympathy for the free thing not being good enough

it's pretty reasonable for them to ask for a CC -- making it too easy to get free compute/bandwidth is opening the door wide for abuse.

..but yeah, everyone wishes they'd have a sane way to halt services if over budget.

Yup, this is one of several reasons I just pay the $5/mo for a similarly specced VPS with another provider.

Off topic, but Oracle cloud's free tier never charges unless you upgrade your account. You can have 5 VMs, 200 GB block storage, etc.

Yeah but they also probably desperately need customers. I can imagine that they have lost a lot too AWS Azure and Google.

Only 4, not 5. Because each instance requires boot volume of minimal 50GB size. And free tier gives you 200GB of volume. I've tried to setup fifth instance but received obscure errors, till I realized that there is not enough space for boot volume. Also there is no possibility to create smaller boot volume or decrease size of already existing volume. Have I miss-looked something?

Also the AWS free tier expires after 12 months (unlike, say, GCP).

AWS has both 12 months free tiers for some services, as well as pretty generous always-free offerings, like the data transfer announcement today. The 1TB/mo CloudFront traffic is now always free, as is the 100GB/mo of data per month from AWS regions to the Internet.

https://aws.amazon.com/free/

Reply on news.ycombinator.com