Avoid Surprise Bills from AWS | Hacker News Reader

Avoid Surprise Bills from AWS

blog.begin.com

64 pointsmacdonst70 comments

Somewhat surprised none of the major clouds off true sandboxed learning environments as a competitive advantage. i.e. We will not charge you more than $100 no matter what. Sure that doesn't work for enterprise stuff that has to be online, but there is a big market for people that can swallow $100 accidents but not 15k. And whoever offers that first will attract the cautious crowd. I've got friends that point blank refuse to use cloud due to the open ended billing.

(Azure comes closes as best as I can tell...their MSDN style credits seem to be true hard capped)

[Invariably this get's the "but what to do about the data...can't just delete it...yes you can. The $100 crowd is ok with that. 30 day grace then delete it...much like anyone with an unpaid bill]

Google Cloud does this with new trial accounts, there's a very explicit step between free and paid services.

>Google Cloud does this with new trial accounts

I've already got an account that has used free credits. So me creating another account to utilise the free allowance again would presumably be not OK

Isn't this DigitalOcean's whole business model, basically?

Yeah. There are lots of hard capped options...just not among the big clouds. Which is fair

I just don't get why the big 3 haven't tried to undercut their competition with this.

It's a very easy way to attract casual dabblers and enthusiasts to your platforms

Arguably, casual dabblers is what gets you increased support costs and does not land the big contracts. I wouldn't be surprised if AWS and GCP _intentionally_ do not do this because they'd rather focus on large enterprises "who are willing to foot the bill"

> We will not charge you more than $100 no matter what. Sure that doesn't work for enterprise stuff that has to be online, but there is a big market for people that can swallow $100 accidents but not 15k

I’m sure this market is actually tiny

And you'd be competing for the price-sensitive (ie worst) customers

Exactly.

Or maybe you want that crowd to go to someone else's infrastructure. And therefore you don't build it, and no one else builds it.

If only there were a way to stop the surprise billing once it starts.

A friend created an AWS account with a new email address for a one-off side project, and wound up locked out of both the email account and the AWS account. But AWS has the credit card, so a 25 cent mystery charge appears every month.

AWS support is completely useless despite years of attempts at escalation, and of course the credit card company can't stop the automatic payments. The "solution" from both AWS and the credit card company is to... dispute the charge every month.

Forever.

That's a lot of worries about 3$ per year.

I think it's the premise rather than the cost that matters. I'd be scared that $3 turns into $300 or something

Or everyone starts doing it. That will quickly add up. Just think of revenue they could get from millions of users for providing no service at all...

Report the card as stolen so it's reissued with a new number :)

I did that and next month new unapproved charge again. What happened is they auto updates stolen card with Postmates and then told me to contact Postmates but I don’t have account with them. I basically now have to cancel credit card every month. I’m just gonna have to figure out how to get new one.

This is because issuing a new card doesn't invalidate the digital token associated with your card. E.g. if you have something like Google Pay, issuing a new card, Google Pay will keep working even though your card details have changed. You need to contact your card provider and get them to delete/renew the digital token.

It's an absurd solution, but they can cancel the card and get a new one, and that will stop AWS from being able to bill them.

This is a good way to go to collections (at least in the US) and have your credit rating harmed. Probably not for .25c but everyone who says just to cancel a card to stop annoying bills is not a good answer.

I've done exactly what I suggested and I've never had a bill sent to collections. YMMV, of course.

In general you have a point (eg don't try to "cancel" your gym membership by using a throwaway card), but as long as you have the right to cancel the contract and you notify the business of such (eg certified mail, in the worst case), then you're in the clear. It sounds like AWS support had already been notified, so revoking the payment channel is totally applicable in this scenario.

Maybe, however Visa has "account updater" which is an API so that companies with saved payment methods can update them to a new number once the old expires or is replaced. They might need to close their bank account if the bank participates in this.

https://developer.visa.com/capabilities/vau

This feature can be turned off.

My card expired and now I'm getting every month a notice that my AWS account is going to be suspended. It's been 2 years...

It won't work if the service provider (e.g. AWS, GCP) has set up the card payment as a recurring payment. All payment networks allow for the «recurring payment» flag (or its direct equivalent) to set to «true» at the time the first payment is made, and the service provider will continue to automatically charge your card account until you explicitely cancel the payment / service contract (sometimes through having to engage the customer service). For example, a local government agency that charges me for the road toll use continues to charge my using a card number that expired in 2018.

It is important to understand the difference between the card number that is embossed/etched on the physical card (or the virtual card number) and the internal card account number. It ultimately boils down the financial institution that has issued the card, but the card account number may pop up on the monthly card statement or elsewhere, and it will be different from that of the issued card number. Many financial institution now hide the card account number from the card user, but it is usually there on the system (new fintech startups might do it differently, though).

Recurring payments are always set up against the card account number, and the card account will continue to get billed, even if the card account has been closed and the cardholder no longer has the business with the financial institution that issued the card – until such a payment is explicitely cancelled with the business. Virtual or one-off card numbers get declined for recurrent payments if the card number is fully decoupled from the cardholder's card account – the payment networks mandate the card issuer has such checks in place. For instance, even if the card number is shielded with a PayPal handle, PayPal will still diligently honour recurring payments and will bill the underlying card.

Most of the time, cards set up as with recurring payment flag on are convenient for the cardholder (card has been lost and reissued, card has expired and has been reissued etc) and for the service provider (fewer enquiries), but there is a sizeable number of businesses (even legit ones) out there that engage in shady practices that have burned or surprised more than one consumer with a nasty letter from collections 1+ year after cancelling a card product.

I had a similar thing recently. As I understand it, AWS uses a digital token for your added a credit card. Contact your card provider to renew/rotate/delete the digital token that is attached to your card. That should stop charges.

Doing that obviously might impact other services for which you use that card so you might need to re-add your card to those services.

Generally tokenization is done by some vault or processor. I don't know how Amazon operates internally, due to their scale means it's probably all in house. But Visa does have token representations and they do also have visa account updater (which keeps visa/merchants in sync for saved instruments .. e.g. Netflix). I've never seen the token implemented, generally just some vault (Braintree, Spreedly, etc). And if you are typing in a 15/16 digit number it's likely being vaulted/tokenized but not by the issuer but by the merchant/processor.

Stripe provides virtual cards so I use a separate one for each online service and can delete a card at a moment’s notice.

AWS offers alarms and actions that will stop or terminate your service based on budget thresholds to solve exactly this problem. Why aren't these sufficient?

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

It's literally the first thing I do when spinning up an instance in case I have a bug or go viral (never happened).

I don't have access to the dashboard because I cancelled my AWS account rather than try to spend more time finding things that still needed to be unconfigured/disabled to get them to stop billing me (the last straw was going to the spend analyzer and it telling me it'd take 24 hours to see what is still costing me money) but:

Can I terminate based on cost? Like "I have spent $1,000 this month in AWS, something has gone wrong just kill everything" (or at least runaway service buckets) or is it just "oh I forgot to terminate this particular EC2 instance once I was done with it it'd be nice if I could just set those rules up in advance"?

Yes.

Step 1. Go to billing and create a monthly budget. Mine is $100.

Step 2. Create an alert: First alert is I get an email when it exceeds 80% ($80) total AWS costs.

Step 3. Create an action: I only have a single EC2 instance running a webserver that is always on. If my threshold is exceeded (say, a million people start downloading my pictures and my IO-OUT spikes), my action stops my EC2 inst via an IAM role action. Boom. Server goes offline instantly, without having to log in (like if I'm sleeping, or drunk).

Done.

Sometimes I get an alert because my usual cost is $35/mo and if a few domain renewals pop up that month, it will spike to $80. Hence the alert at $80 and action at $100 threshold.

And I can use any kind of metric: IO bandwidth from downloads, RDS bandwidth for too many queries, if I had elastic instances, limit the # based on cost. It is completely flexible. You can terminate too, but I only have one, I don't use elastic pools to dynamically allocate.

I don't get all the fuss, it is quite a simple service. Maybe it doesn't scale well for huge operations and that is the problem cuz i'm not a power user or company?

Is step 3 literally "create an upfront action for the single EC2 service I have configured"? If so yes then the problem is scaling, 1 thing by 1 person created in 1 day in AWS is pretty easy to manage even without this feature but 1,000 things across many service buckets where 1 is something like a runaway suspended machine in a region you can't find when it should have been terminated and you don't know what needs to be terminated you just can't click a button and see it rather you have to go down each breadcrumb trail of billing buckets that look odd and jump between portions of the interface trying to cross track it is is where it turns into a disaster.

On the corporate side it's a project where a team tries to go through everything and hopefully people have stayed in their lane on things they configured in AWS so the SMEs can just check their stuff and find it quickly. On the personal side it's a lamentation there isn't just a "nuke all" button beyond permanently disabling your account completely.

It can be a nuke or a surgical scalpel, e.g., contour traffic rather than taking down your entire site. And it is scriptable: any IAM role can be programmed into an action.

> If so yes then the problem is scaling,

Come on, man: you can't bash AWS if you don't even know how it works!

I'm addressing all these sob-stories of poor college students suddenly getting hit with $1000 bills for using lamda the wrong way, not a Series B startup with $5MM in the bank 20 employees and a billion CPM on their webapp.

I don't mean can you scale it down granularly or stop the service completely I mean can you say "when $100 disable everything in this AWS account that will generate billing without having to specify each thing individually in a rule". Snapshots, backups, IPs, instances, etc". It's not a matter of knowing how these things work it's a matter of finding what you're going to be billed for tomorrow because it is currently running - that's what's hard.

> Come on, man: you can't bash AWS if you don't even know how it works!

It is possible to understand how AWS works and still run into problems trying to scale AWS billing. This may not be apparent in a single ec2 instance setup but that doesn't mean the reason you see the complaints so often is everyone else are just idiots.

In my case I didn't lose 1000s or anything on my personal accounts more like 40 bucks by the time I just closed the account rather than wait 24 hours to track down the last thing in spend analyzer. It was a precanned product demo script for a cloud security product, first install went wrong and needed to be cleaned up manually but it was hard to tell what actually ended up staying vs not, especially since I didn't define the architecture ground up manually.

Note this is separate from "I didn't know if I clicked create 1000 GPU training instances it would cost a lot" though that would also be covered by an upfront monthly limit too I suppose.

Alternatively: imagine how quickly the UI would be fixed if the difficulty in finding how to create a new billable service were switched with the difficulty of finding which billable service is causing overruns.

A solution for the hobbyist who can't foot a 15k bill was asked, and given.

It's the difference between a limit order, and writing a script that monitors the price of a stonk and submits a market order.

Well, limit orders aren't guaranteed to trade on the spot either. Any reputable brokerage will warn you of this when you create one.

There will always be lag, but I've never tested how quickly AWS monitors trigger under extreme load.

The blog post says surprise bills rarely happen but nearly every company I've worked at that used AWS had surprise bills and had reasonably large projects to reign those bills in once they noticed the bills.

This is such a large problem for AWS users that there are multiple SaaS systems that help people manage their AWS bills.

If you're using Datadog with its AWS integration, you can pull in your billing metrics and set up anomaly detection monitoring. If you set it up as a multi-alert, Datadog can event alert you on the specific category that has a billing anomaly.

You can do that with native CloudWatch monitoring, too, but it doesnt work well with spikey charges like up front reservations and things billed at the beginning of the month

Technically they are "anomalies" but there's no human interaction required so you get false positive alarms. It also doesn't work well beyond a certain $ amount since the error bars get pretty big

Is DataDog better with that?

With DD you can set up monitor downtimes in advance of expected anomalies. Also, at the beginning of the month costs can drop suddenly, so what you can do is set up separate 'over' and 'under' threshold monitors, and then set up downtime on the 'under' threshold at the beginning of the month.

Use one opaque black box of costs to get clarity on another opaque black box of costs. I don't recommend this.

Datadog very quickly adds on costs for things you have no idea you need or want. You have an additional "host parametrix double-channel provider" and it's an extra $200/month per 2000 items. 2000 items of what? I don't know but you are at 1900 now and there's still 10 days to go in the month. Also, to manage those 2000 items, you need 2 additional pipe funnels so the data is churned to the right settings.... and so on.

The sales team signed you on $60/month and now you're at $2189.48 and counting. Where the fuck is all this money going? you say to yourself and the inertia takes over and you close the Datadog tab.

I really reach to use DD but I’ve heard a ton of stories like this and they scared me away.

It's not like hand-rolling the equivalent services that DD provides is free.

> now you're at $2189.48 and counting.

This is a wonderful bargain compared to the expense of hiring a full-time engineer (or more) to provision and maintain all the infrastructure that DD replaces.

Can someone just make “AWS insurance”? Honestly seems like a service that some people might pay for.. like pay $100/month to have some service monitor and guarantee that your services will be scaled back or turned off if you hit your quota.

Or just setup regular alerts / emails from AWS...

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

You can still rack up a huge bill if you run a public web service and don't monitor the alerts 24x7.

There are so many unlimited liability services that are extremely difficult to avoid as a business.

I presume the primary issues is that service suppliers don't want to implement the systems to actually measure realtime costs properly (due to implementation costs, and balance sheet liability avoidance?).

For example we wished to buy an IP phone service, but to avoid the high cost of getting hacked I wanted a way to limit our liability. There were not many choices. We choose a solution that was a prepay dollars-per-month-per-line system where we still signed unlimited liability: but at least it was more likely the cost would be controlled by the provider to the prepay amount (unless limit was lifted by crackers changing account limits). We were also careful with website passwords, and we paid on a business credit card for a second layer of protection.

We are using a bunch of cloud services at the moment such as AWS, GCP, Snowflake, Databricks.

These are all fantastic and I’m glad that we have them, but at the end of the month it feels like gut punches when the bills roll in.

Between the infrastructure as code, the auto scaling and the weird credit based systems, it feels totally out of control even at micro scale.

If I ever held the budget for a large organisation again I wouldn’t sleep at night with the amount of financial risk.

Cloud cost control should definetly be a growing demand in the future.

Step 1: Avoid AWS.

This was my first thought. Understand that it's a financial risk to work with AWS at both small and large scale.

I have a personal AWS setup that has no costs, at this time. I've run a business account at 50k+/mo. I've never had an account suspended or lost my credentials and those stories scare me a bit.

After trying to set up Fargate and it only sending my logs to cloudwatch 25% of the time (Id deploy and get no logs, try again two to three times literally just clicking the button in the web console again and then it would work), if course AWS charged me for every . Two days and a $150 bill later I got fed up and moved to DigitalOcean App platform, where I fixed the issue (was a bug in my health checks) within ten minutes and a monthly bill of $10. That day I decided I will never again use AWS unless I can pay someone who really knows what they’re doing to do it for me (and I have a good reason to actually want to use AWS). My life has been much less stressful since.

How much does a domain name like "begin.com" cost, and how do you get a good domain / brand name for your startup (without calling it "PurpleKerfuffle" or something unregistered)?

Is there a market for this? A strategy?

I'm trying to get something good, but I keep getting back figures in the millions of dollars. Even ".io" domains are $100k. It's ridiculous. Am I doing something wrong?

Either you are the squatter, you pay the squatter, or you dodge the squatter. It is what it is.

Exactly all the same principles as the real estate market now basically.

Some of us People of Age remember when getting a good domain was easy before squatters :)

And yes its hard now. In real life you can have the same business name in two different business areas as long as it doesn't cause confusion. I think we should get rid of domains all the way, I should be able to register any text and make it DNS resolve.

Only answer for having squatters is to make any "domain" buyable so they can't get all the good ones.

> I think we should get rid of domains all the way, I should be able to register any text and make it DNS resolve.

> Only answer for having squatters is to make any "domain" buyable so they can't get all the good ones.

How's that any different from the current domain name system? There will still be only one owner of the text "google" or "begin".

Because then there is also g.oogle, go.ogle, goo.gle, goog.le, googl.e. Or my.google. Or whatever, more options.

Running my own development service, I usually manage to find unique names for people all the time. If they really need a specific name, it's not always expensive to buy it from certain (legit only) places...

We don't use AWS for most of our solutions, we still use traditional (fixed price) hosting. AWS changed hosting into a utility-based service that can balloon way too fast, and they make TONS of money off of unsuspecting clients, especially government clients. It's wild to see their bills at times for even simple web sites that don't get tons of traffic. On-Prem services are better sometimes, but no-one wants to hear that.

How much was that domain name?

Lol, we get that question all the time. One day we got offered $17,000 USD for the domain and it wasn't even the best offer that day

So if that wasn't the best offer for the day, what makes that offer stand out to be mentioned vs the highest offer you had received that day?

I dropped AWS like a hot potato once the alternative solution for hosting large media files became feasible. I was sick of their bills.

And now with dedicated servers / storage / decent bandwidth so affordable it would take a tectonic event for me to go anywhere near cloudy stuff for any service. Well I did some Azure because client required it but that is their money - their problem.

This is just an ad, why is it being upvoted so much?

Because the title touches a pain point that's existed on AWS for over a decade, and that also exists on every cloud provider. These multibillion dollar operations refuse to implement an option to set hard budget limits even though their customers want it and would benefit from it.

I have heard nothing but negative things about aws billing, such as overcharging and surprise billing. glad I do not use aws.

Move away from vendor lock-in.

Is there a way to keep the lambda concurrency and perhaps introduce an x% error?

I got a very surprising bill from GCP. I used bigquery to query the google public patent database[1]. I thought I was using enough predicates for it to effectively be a point query (using patent ID, etc).

Turns out each single query was a full table scan of the 4TB dataset and cost about $5 each! I luckily stopped after 4 queries once my curiosity was satisfied, but I could have easily spent hundreds of dollars. It wasn't until some time later I saw my bill and realized what happened.

They did do a "one-time charge reversal" for me. I didn't really feel satisfied with that response. I wanted them to admit such datasets are a huge liability for users.

1. https://github.com/google/patents-public-data

They should have just billed you - they reversed the charges (for your own issue) and you aren't happy.

I think folks are used to being pretty catered to.

And yes, being able to play with 4TB databases in the cloud and huge GPU instances is a "liability" - you pay for what you use.

Google actually gives you a lot of data about jobs folks are running - check out

https://cloud.google.com/bigquery/docs/information-schema-jo...

and total bytes processed if you want to do estimates of a per job cost etc.

Reply on news.ycombinator.com