$45k in AWS charges due to account being hacked to mine cryptocurrency | Hacker News Reader

$45k in AWS charges due to account being hacked to mine cryptocurrency

twitter.com

146 pointstomduncalf134 comments

So, I have a personal AWS account, and setting up cost controls seems like a good idea. Let's try it out.

"Welcome to AWS Cost Management

Since this is your first visit, it will take some time to prepare your cost and usage data. Please check back in 24 hours."

My "cost and usage data" is that I have an S3 account costing ~$10 a month containing my cloud backups of my family's high-priority data, for maybe the last 18 months. That's it. I just reloaded in case 24 hours was some sort of pessimistic estimate and they actually had it done in a couple of minutes, but the message persists.

WTF.

I can set up any of dozens of services in thirty seconds and spend thousands, but it's going to take 24 hours to understand ~$100 in costs over the course of the last year? Someone could read aloud the entire history of the account, every S3 transaction I did, in that amount of time.

I can really feel the love here.

Not to mention AWS only updates billing data 2-4 times per day. So even if you do have this enabled, on an accrued cost basis you're going to have to wait 6-8 hours to know about it from Cost Explorer.

AWS doesn't benefit by providing excellent cost controls and visibility into spending.

I'm not saying that neglecting this is a conscious, cynical choice. It may just be that there's simple no incentive to build it in the first place. A self-motivated engineer or team manager is probably not going to say, "I know! Instead of working on the next bit of shiny exciting tech, I'm going to work on cost controls!" Though sure, it could also be deliberate.

That's not true at all. Businesses want to reduce risk. Providing cost controls assuages a client's fears. If you're picking between two clouds, cost is a concern, and cost risk is a big factor in that. Would you put your career on the line for a riskier cloud? That's not only cost, but also security, privacy, legal risk, compliance, etc. Clouds spend a great deal of resources to improve all these things, especially for highly-regulated industries which constitute a large part of the potential customer-base: health care, banking, government, polluters.

And yet AWS has awful cost controls. My interpretation of why they have bad tools in this area might be wrong, but what is the alternative explanation? Incompetence? Certainly they must know it's a concern for a subset of their users.

You also may not be cynical enough here: Lookup the Amex tax avoidance strategy that they pitched to companies for a while before a whistle blower tipped off the IRS [0]. Or pretty much any fraud that originates at a high level in a corporation. Companies make risk/reward decisions about doing bad things all of the time. ::ahem Theranos::

[0] https://www.wsj.com/articles/amex-pitched-business-customers...

From a few different devs at AWS I have heard a couple of common explanations:

- Billing code/codebase and everything around it is (was?) a big crapshow and people are naturally reluctant to stick their hands in something so critical and hard to adjust (and equally reluctant to approve such projects).

- Some teams believe that customers don't want such a feature and may have been explicitly told that they don't want that feature to exist. The latter part seemed like hyperbole to me but I don't have the context they have.

naturally reluctant to stick their hands in something so critical and hard to adjust

That pretty much lines up with my comment about engineers that would prefer to work on some bit of shiny new tech that's more glamorous than cost controls, and my general sentiment that AWS simply doesn't benefit from pursuing these changes.

I imagine it's also the fact that you are rewarded for working on complicated projects that impact the bottom line positively.

Driving this billing project is going to be thankless work, and at the end of it Amazon will lose money.

It's a lose-lose.

> - Some teams believe that customers don't want such a feature and may have been explicitly told that they don't want that feature to exist. The latter part seemed like hyperbole to me but I don't have the context they have.

I have heard such reasoning many times to get away with all sort of BS.

The argument is often impossible to refute, since you don't have any customer at the meeting and the feature is on the idea stage. So I guess it is some sort of argumentation fallacy, referring to a non-existing person?

So I guess it is some sort of argumentation fallacy, referring to a non-existing person?

The person might exist, but either way I'd say the fallacy is an improper argument from authority, in two stages: First, AWS is claiming authority on the matter and citing other customers as their evidence. That evidence is an improper appeal to authority as well because the expertise (or even existence) of the person(s) behind it cannot be validated. Sort of like a celebrity endorsement where you don't even know who the celebrity is-- e.g., An unnamed person who endorses a basketball sneaker but you don't even know who if they play professionally and actually use the sneakers themselves.

For that matter, even a proper appeal to authority (area of expertise is relevant to the topic at hand, expertise had been validated, there are no conflicts of interest like getting paid for it) are not very strong evidence for a claim. They are indirect evidence where the person is saying "You don't need the data, I've reviewed it and you can trust me".

The likely explanation is that there hasn't been enough pressure put on them to improve these things. They care primarily about large customers, and they already have plenty of ways to resolve cost issues, and usually sign custom contracts. If GCP or Azure has a significantly better answer to this problem, AWS might start caring, but I don't think they do. (I work at GCP.)

I think that lines up nicely with my general point that AWS simply doesn't benefit much from putting resources into this area, although some C-level folks might be enticed by shiny marketing materials about "Best in Class Cost Controls & Monitoring"

For other option, it does seem like GCP (I don't know much about Azure in this) is a little better in this area. Admittedly, I'm speaking from limited experience playing around with free tiers in both, along with the stream of stories that we hear about AWS.

I think it's also fair to point out that cost controls may not have helped much in the example of this particular article: If your account gets hacked, presumably the hacker will also have some access to remove cost controls.

Realistically, this wont be a priority until Gartner has a category for this and places all these cloud players on a ranking scale. That sort of thing seems to really move the needle for execs.

Ah, yes. I always give a little internal eye roll when a vendor highlights their placement in a magic quadrant.

If you are a small team and worried about cost, which everyone usual is, the best bet is not to use AWS at all and go with something like DigitalOcean or Hetzner where the pricing tends to come with less surprises.

And setting up two-factor authentication is far easier.

That seems reasonable, although the argument for AWS is usually that it can scale much more easily. If your primary use case is something like EC2, that's probably a non issue when choosing DO unless you have very specific needs for your instances.

> AWS doesn't benefit by providing excellent cost controls and visibility into spending.

Except you can create a budget that is based on individual services or usage instances. And you can view each service's cost in zoom-able billing statement.

What information are you unable to find about your bill?

(NOTE: I'm not an AWS shill. Every time I see AWS billing threads I'm concerned I'm missing something obvious and putting myself and my biz at risk, so I drill down on posts to make sure I know what I'm doing.)

www.privacy.com

The billing system runs each night.

This.

Many changes to Google Workspace take up to 24 hours, as well. It always blows me away when I make a tiny change and get that message.

This is absurd. I just logged into the billing alerts console. I set up a total billing threshold in 90 seconds:

https://console.aws.amazon.com/billing/home?#/budgets/overvi...

What on earth are you getting at?

You can setup whatever alert threshold you want, but it doesn't stop charges from accruing.

What is needed is a way to say "I only expect charges of $10 or less per month. If charges go over that, disable the account until I can get in there and find out what is happening."

That can also be done. I can shut down any specific or cluster of EC2 instances if any one exceeds a $ limit, or all of them. The resolution is up to me and I immediately stop getting charged.

If I have lost ROOT access, as the OP did, nothing can save me: all alerts and thresholds can be shut off by the hacker. Just like any other system.

I saw that part about setting up quotas and actions when budgets are exceeded. I did setup an email alert, but the quota thing was extremely complex. There was a list of service quotas a mile long just under the EC2 section. Some of them were automatically set by AWS, most of them were zero. This is clearly aimed at companies with a huge AWS infrastructure investment and the staff to manage it, not some schmuck individual like myself that just wants to not get a $45K bill.

Again, the $45K was due to a hack of root credentials. Not AWS being shady. If anyone got access to root on your machine you'd be screwed.

If a total shutdown when aggregate monthly budget is triggered is insufficient, and fine-grained quotas are too complex, you probably shouldn't be in the cloud because Azure and Google are equally complex. Or do you like another cloud-computer service better than AWS because their budgeting quota systems are better? Please share, I'd like to know what you use.

> If a total shutdown when aggregate monthly budget is triggered is insufficient

I never said that. In fact, that's what I'd prefer. I don't use AWS for anything critical, only for testing, so I'm fine with my account getting temporarily disabled or all requests returning error codes.

> Please share, I'd like to know what you use.

B2. B2 has a 10GB free tier for object storage and returns an error code if that limit is exceeded without a credit card on file. They have limits (called Caps), like AWS, but B2 apparently automatically sets their limits low, because when I checked, my limits are $1, $5, $1, $1. Pretty sure I never set these. They will email or text when limits are exceeded by 70% and 100%. As far as I can tell, they don't have a hard limit (return error codes) once a credit card is on file like they do with no credit card on file, but I think that would be a good thing to add.

Have you ever visited the page before? The message explicitly says that because this is my first time they need 24 hours to set it up.

I can only report what happened. Why this couldn't be preloaded for my teeny tiny account and its probably on the order of dozens of kilobytes of data is something you'd have to take up with Amazon, not me.

Yeah good luck with anything other than e-mail though, if you want SMS notifications you have to go through setting them up through a layer of indirection through AWS SNS messaging.

And it'd be nice to just have a $100/month killswitch on my personal AWS account and I have no idea how possible that is with their actions (which seems to require configuring IAM accounts). I don't run anything "production" there and I'd be happy to have all my shit suspended if that dollar figure was exceeded, but fuck if I know how to do that without taking a week to catch up on the past decade of how to manage infrastructure with AWS.

It may be simpler to just suspend my AWS account entirely. I think I only have some ancientsauce mysql backups on S3 which are costing me a few pennies a month.

This is exactly the scenario that people post in every thread about AWS failing to implement budgets and hard limits, and every time people always say "but AWS will just forgive the fee!" as an excuse for AWS being so user hostile.

So far, this time, they haven't. I hope they do for this guy's sake, but to not have that as an official policy, or to implement a tech solution, is beyond terrible.

Yeah, happened to me with an account that:

* Never used anything above the free tier

* Never had a credit card attached

* Was only ever logged in from the UK

* Hadn't been used at all in 3 years

Naively my root credentials were the same as my Linkedin details (which were all leaked), and one random Sunday, someone from Brazil logged in and spun up 50+ GPU instances across multiple zones. Amazon let that go for around 12 hours before they sent me an email asking me to check that my deployments were working as expected. £2,000 in just those few hours.

I did get the fee waived (though they took two weeks to finally decide this), and I accepted that it was my fault for reusing passwords and not enabling MFA (which IIRC may not have been an option when the account was opened) but I was very critical of Amazon for not preventing expensive nodes from being spun up when there wasn't even a card attached to the account.

I really have issues understanding the "I did re-use my password for a service that might charge me an indefinit amount. But i still think, it's the services error".

I mean, it could "never" happened if you just thought of a better password for your AWS account. Why blame AWS for that? All the preventive things AWS could implement wimm help the few users who didn't get that they should good passwords and hinder all the users who "know" what they are doing and just need some machines up-and-running. IMHO.

Maybe it is more profitable for them to not have hard limits for legit cloud uses and deal with hackings in a way to just forgive. I am pretty sure they have some calculations done showing what way is more profitable for them.

More profitable for them but user hostile

> AWS failing to implement budgets and hard limits

The problem isn't that AWS does not let you manage limits, because they do, to some extent: https://docs.aws.amazon.com/general/latest/gr/aws_service_li...

The problem is that if someone has your root credentials, they can just change these limits and go to town.

Can't the guy contest the charges with the CC company? What recourse does Amazon have beyond locking the account?

AWS bills monthly, so his CC hadn't been charged yet. It was just the number that showed up in his AWS console.

In previous threads, people have said that AWS send unpaid bills to debt collectors.

Actually discussed the problems around dynamic billing on HN a while ago, here's my comment on the topic, alongside some other platform recommendations, at least as far as VPS hosting is concerned (which is suitable for me since i run a lot of software in containers, but perhaps not to everyone else): https://news.ycombinator.com/item?id=29339477

From there, a discussion continued, but the problem of what to do when some sort of resource allotment is exceeded, is still not solved.

Personally, i think that static allotments (e.g. you get a VPS with X bandwidth per a month, where the port speed will be decreased 10x until end of billing period in case it's exceeded) are the way to go for people working on non-critical projects, where downtime or degraded performance is preferable to not being able to buy food or afford rent.

Though with some platforms, you still need to protect yourself, either with virtual cards or other means, since there's no guarantee that cases like those in this post will be noticed and the fees even waived by the corporations. Of course, the legal aspects of all of that are somewhat unclear. All the more reason to look for solutions with simpler billing, or at least billing alerts in lieu of that.

I'm still absolutely blown away we tolerate an industry model with uncapped cost. The fact that anyone will even do business with a cloud provider built like this absolutely blows my mind, like every CFO all collectively lost their marbles.

Fortune 500 companies have literally just decided handing blank checks to Amazon and Microsoft every month to make their technology needs disappear was good business sense.

I wonder if they have different contracts from most of us?

How health care works. Show up anywhere. They can more or less charge you anything they want, destroy your credit if you can’t pay.

Of course the conversation is usually about how to get someone else to pay the bill instead of get costs under control.

The American dream.

"the conversation is usually about how to get someone else to pay the bill instead of get costs under control"

Unfortunately most of us Americans cannot comprehend this fact or are too helpless to do anything about it. I love the folks who tout their awesome employer based plan without really thinking about the insane costs just because they are not the ones paying most of it. I also love the folks who just think a total Universal Healthcare like Medicare for All is the solution which honestly I am not totally against but only if the costs are first regulated in any way.

Universal health care in itself is a driver for regulating costs if it’s implemented as single payer. The idea is there is a monopoly on supply of patients, so if you want to practice, you have to accept the billing rate the payer (govt) deems necessary. It’s like price collusion but done in order to depress prices.

I am willing to try anything that what we have right now in healthcare BUT looking at Medicare by itself, I am a bit wary of Govt's involvement completely. I think we should cut out the middleman mafia a.k.a Insurance company from EVERY visit and instead let care providers compete in the market. i am ok for a nephrologist to charge $x more than the next door nephrologist as long as their service and care is better and let the patients decide that. I do agree that Govt. should step in to probably regulate price gouging so that no one can charge me $500 for a toilet paper roll.

Trying to hold on to the small government ideas and remove the government along with health insurance companies from the equation is a non-starter. As is saying that you want competition, but limited government, but also no gouging.

Competition is how your stop gauging.

Competition only works when you understand the service process, and can make a decision to go to another provider with little cost. When they bring in a medical provider or have to perform another procedure while you are unconscious, you can't defend against gouging.

There's actually a term for this - monopsony. I've only ever heard it used in the context of universal healthcare.

The theory: government sets the prices because it controls the supply of patients

The practice: you pay for a news article about a grandma unable to get healthcare. The government reacts by ensuring that all healthcare is paid for - with no cap to cost. After all, every life is sacred.

COVID-19 response showed us.

> My mother is not expendable. Your mother is not expendable.

> We will not put a dollar figure on human life.

https://mobile.twitter.com/nygovcuomo/status/124247702908329...

>I'm still absolutely blown away we tolerate an industry model with uncapped cost

Try reading the contract you have to sign before receiving medical care :)

Water is the same deal. If the water pipe going between your house and your garage leaks 30 olympic swimming pools worth of water deep underground, your utility company will be more than happy to set you up with a payment plan.

AWS isn't the only example of giving blank checks to some vendor...

Although it'd be hard not to notice before the leak is even halfway through one swimming pool worth. On AWS it's fairly easy for the charges to rack up for quite a while.

(Also, at least around here if the water agency notices abnormal flow for a long time they drive by and if they see a leak they shut off the water. Happened to me once when a sprinkler pipe blew out.)

Not in my case. Leaked about three Olympic pools of water and we have zero clue where any of it went.

Exactly! This is the one reason I refuse to use AWS and am constantly surprised so many people are willing to have this risk on them. I'd rather get a VPS with DigitalOcean and know that they cannot charge me tens of thousands if somebody hacks my account.

I have a theory that this is why the cloud is popular with big businesses.

Every big business has an in-house data centre, which is operated as a cost centre; success, for a cost centre, means delivering service as cheaply as possible. Providing an easy way for developers to spin up new databases? That would practically be an invitation to increase costs - it's the last thing they should be doing!

So when a developer from that big corporation uses AWS and encounters an environment that makes it easy for them to spin up a new database, it's a big productivity increase.

I absolutely understand why in-house IT and dev folks like it: It literally removes the ability for budgeting people to tell them no. And the cost of new uses starts small and is granular, so by the time finance folks see it, it's already done and a critical component and no longer optional.

It literally removes agency from financial folks and gives it to tech people.

This is good and bad. It's good if you have competent tech staff. Right now my org is battling for basic resources like gsuite and okta licenses for their new hires because the CFO is complaining. It's absurd.

If you have competent tech staff you can inhouse it on any of a dozen hypervisors for roughly the capital cost you would spend in 6 months on cloud.

Every year on the cloud can roughly cover the expense for double that capacity deployed on prem.

Complaints of "operational costs" generally assume a woefully incompetent staff.

This has happened before as a tech manager in different companies. When looking to implement some new technology (say, a new Cassandra like database) I have two options: either a) I choose ScyllaDB, and have to go through all the "additional vendor" internal process, that with ISO27001 gets terribly bureaucratic; or b) I start using DynamoDB or Keyspaces transparently and let the current AWS bill increase.

For me it has always been a no brainer to avoid the bureaucratic pain.

And despite your misgivings, businesses in every industry have now been on AWS for years without the doom predicted in every thread.

There's no doom here, just exactly what we are seeing: Businesses running on increasingly thinner margins, except tech companies, sitting on wild profits from this business model.

So big tech is vampirising all the other companies and are responsible for our economic downturn? No.

> And despite your misgivings, businesses in every industry have now been on AWS for years without the doom predicted in every thread.

Well the doom does sometimes happen, perhaps not well publicized. I've been in one startup that ran out of money where the astounding AWS bill was part of the demise. And in another startup where the AWS bill hindered growth since it took such a large slice of the funding.

  > I'm still absolutely blown away we tolerate an industry model with uncapped cost.

How could the cost be capped? Fortune 500 companies run their global infrastructure on AWS, so AWS has to support these huge use cases. If you've set "spending limits", an adversary that has your credentials can change your limits to those same limits used by the F500 companies.

> If you've set "spending limits", an adversary that has your credentials can change your limits to those same limits used by the F500 companies.

The way I have seen this being operazionalized in AWS is by Finance having the exclusive access to a "parent" AWS account controlling billing (and nothing else), and then providing sub-acconts to Production, Development, Data Science and other technical stakeholders.

It's not uncapped cost. Almost every service has a default limit to how many instances/butckets/etc that you can create. All of that creates a monthly cap of somewhere likely south of $1B. As long as you have that much cash on hand you shouldn't have to worry about getting ruined.

This happened to me back in 2014, albeit with a smaller bill of around ~$10k.

I was pretty inexperienced and I accidentally committed my Amazon credentials to git and went to bed.

When I woke up I had a handful of emails from Amazon alerting me to the situation. Back then $10k was almost a third of my yearly income, so it was a pretty traumatic morning say the least hah.

When I spoke with an Amazon representative they were very nice about the situation and dropped all the charges, since it was obvious what had happened. I'm sure that's what will happen in this situation, especially given the social media attention.

AWS really needs a hard limit of $50 by default. Any user that needs more needs to explicitly set the new limit only after some type of 2FA preferably an OTP verification. Most of the times these hacks are often due to leaked API keys.

These soft limits and email alerts are easy to miss and a lot happens by the time you can stop it.

Agreed. Things like static IP addresses and storage incur costs over time regardless of whether they are in use. But a limit would at least prevent new resources like this from being provisioned.

Cost management at AWS is an absolute mess. And I believe it is done on purpose. I was tasked with finding out how much a virtual machine was costing us. It didn't have tags. The task was very literally impossible.

I remember being very new to all of this (still am) and being absolutely petrified that as I was exploring an Amazon stack that I'd unintentionally rack up thousands of dollars of charges for an honest mistake. There was part of me that felt so much better about using App Engine at the time; in that case, I had full confidence that the worst I could do is have the app I was building just break if I got to a certain quota limit. That put my mind SO MUCH more at ease. The risk of an accidental cost overrun has kept me from learnings/using AWS to certainly a detrimental degree for me, but the fear of something like this happening is very, very real.

Disclosure - I'm Co-Founder and CEO of https://vantage.sh/

Hopefully the downvote brigade doesn't come after me as I'm truly just trying to share this as I think it'd be helpful in the context of this post and I imagine the majority of folks on HN would fit into our free tier...but this is a huge reason why people use Vantage (linked above)

We profile all resources by hitting List/Describe service APIs in your AWS account and automatically apply rates based off of that. So essentially you get a full inventory of everything in your AWS account and see exactly how much its costing you per hour and also if it were extrapolated out for the full month. People tend to use Vantage as a companion to their IaC that's provisioning things to gut-check from a cost perspective.

Your product doesn't help this situation at all. The problem is people want capped spending for unimportant projects (just cut off service rather than accruing charges), which can only be done by AWS. You're just providing reporting assuming no additional charges.

I've had two experiences with this. Once due to a credential leak through a public GitHub Gist (PSA: Gists are public). Once through the failure to delete a resource with premium IOPS for a couple months (PSA: It's easy to leak resources when developing Terraform scripts on AWS - remember to clean up afterwards.)

In both cases, we received a partial credit. Amazon expects you to setup cost explorer and budget controls before doing anything else. These features must be manually activated on every billing account. It's kind of maddening actually that even turning on Cost Explorer takes 24 hours.

PSA: Turn on Cost Explorer immediately after creating a new AWS account and don't give out API keys until you've setup budget limits. If you join a new company in a role where you have access to the AWS billing account, make sure the cost and budget features are enabled.

Doesn't AWS let you set a monthly limit? I only see a way to set spending alerts, not a hard "do not spend more than X" limit.

There's no way on earth I would use AWS on my personal card unless I could set a limit.

I wouldn't sleep.

No, they explicitly refuse implementing hard limits for obvious reasons in spite of many users asking for it for years.

To the author's point - if you have a strange charge on your credit card your credit card company will reach out to you and let you know. AWS should have similar controls - if your spend gets out of control within a X amount of time period they should reach out to you. Many of the CFM platforms have this as part of their suite of tools (usually called anomaly spend or something similar) and AWS should offer some native tooling.

AWS allows configuring cost alerts via AWS Budgets service.

https://aws.amazon.com/aws-cost-management/aws-budgets/

Yes - that's an opt-in service. IMO it should be on by default.

If a hacker can access your AWS account?

Does it matter -- what AWS limits were imposed?

Those 'real' limits would have to be set by the card issuer. Not AWS. Otherwise trivial for hacker to change the limits.

Amazon should notify you if the limit is changed. You should also be able to set non-editable limits. There's no way I, as a private individual, would spend 45000$ on AWS.

>Amazon should notify you if the limit is changed.

Again if a hacker was to access your AWS account with root this feature would be disabled and you would not be notified. Someone accessing your AWS is fundamentally the problem.

lol @ non-editable limits

Card limits come much later. Usage comes first, then the bill (end of the month).

Even if AWS had a hard spending limit, if an attacker has your root credentials, couldn't they still just increase the limits?

To me, the most outrage thing is AWS is charging $45k to mine $800 worth of XMR. Using dedicated servers, it would cost maybe $2,000-$3,000.

Doing it yourself makes things, faster, better and cheaper than Amazon.

Faster? I really want to know how you can have servers, with all the required networking, and all the associated and easily accessible redundancy ( multi-DC or multi region) faster than a few clicks and max 5 minutes wait time. Not to mention the managed services, how long does it take even for an experienced SRE, to setup a production-ready (with backups, monitoring, HA, etc.) RabbitMQ/database/whatever cluster?

AWS is much easier to use than a DIY infrastructure, unless significant resources are invested in that DIY infrastructure.

Sorry, didn’t mean you get your setup faster, but that you get faster setups. I had an inkling that would be misunderstood.

Yes you can spin up a crap fast on amazon. But it’s crap and you will pay an arm and leg for it.

I don’t care about “easy” as much I care about sustainable and cheap.

> Yes you can spin up a crap fast on amazon. But it’s crap and you will pay an arm and leg for it.

Amazon RDS is everything but not crap. The great value-add that RDS provides is that they take care of all the tasks you'd normally need many days of setting up, evaluating, regularly checking they still work: replication, automated failover, backup/restore, version upgrades.

I believe this is more by design than anything. AWS makes real money from the big guns where they are spending 100s and 1000s of dollars per month (may be even a Million I am sure). If they put limits, it could hurt their business with these big guns who usually have stricter controls on their end and are not leaking keys or don't have 2 factor Auth enabled etc.

The challenge is for the smaller guys who are just experimenting with AWS. I see a lot of these stories where they haven't even used the account much or at all and then get hacked.

Of course AWS can impose limits but they clearly don't want to. They would rather refund these cases whenever they feel like.

I'm certain our monthly corporate bill for amazon is high 6 figures if not 7

I have contracted with several midsize B2B outfits who hit low-to-mid 6 figure USD billing per month for AWS. Surely the biggest players are spending millions per month. At that point, you'd expect Amazon to be much more accommodating if a billing error or exploit were to occur.

The large cloud companies seem dead set against creating any type of hard billing limit. Just let me set a magnitude of charges I can ever expect from the relationship, rather than unconstrained open-ended liability.

We all know it would be straightforward to implement. Even if the time to shut down systems creates some overage that the provider is unwilling to eat (although I can't believe it would cost more than continually refunding these bills), then it would be perfectly fine for "$200 hard limit" to mean start shutting down systems at $100. Just let me limit the damn magnitude!

But since they've all decided it's good business to be sticking users with bullshit bills, we're left to create our own. I've got LLCs on the brain, so I'm wondering if running one's cloud expenditures through an LLC would work.

Besides the paperwork overhead, the biggest worry would be piercing of the corporate veil for being undercapitalized. But I hope it would be hard to argue that a company was undercapitalized because it lacked the ability to pay for unexpected expenses that it did not initiate.

Does anybody know the legal and practical issues with signing up for these cloud providers through an LLC? Even if you're already operating as a company (eg startup), it would still be prudent to sign up through a separate subsidiary. At the very least an extra layer would make for some red tape they'd have to cut through to press the debt.

How are these companies allowed to provide services without direct customer support. We all understand they don't want to pay for the labor, but that can't be an excuse.

We are a very large AWS reseller and this drives me bonkers with EVERY Cloud, and I've written about this countless times.

There is almost no way what so ever to limit risk and it's crazy.

Cloud operators state that it is because they don't want to shut down popular services or similar - and to be honest, I get this argument - but, it is just awful in practice.

Nearly every cloud has cost report delays of anything up to a month for some services and it's just a nightmare.

I kind of agree that if you have a service costing £5,000 a month - you shouldn't limit spend to £5,001 - but, you should be able to limit to £7,500 - enough room to burst/grow, but not have risk!

All clouds could learn a thing or two from telecoms and rate limits... e.g. if you have services that when not in use have a fixed monthly limit - e.g. storage - then take that out of monthly limit and then use the excess for variable costs.

e.g. storage costs £500 a month, compute costs £200 a month... if a client sets a budget of £1,000 - then deduct £700, maybe ~£100 extra for idle transactions (if in a cloud/model that charges), then have prepay of £200 for "extras".

I just see "the clouds" do AMAZING new features - all this crazy and cool AI bits, yet they can't get basic costs correct - it's crazy in my mind.

I just went to set up a hard limit on several accounts. There is email notification. The only actions stop individual services on individual Amazon server locations.

There is no global "DROP DEAD" action freezing my account and my liability to Amazon.

My ideas for fake notification addresses and their purported actions are likely against HN rules. I'm just going to leave Amazon AWS.

I feel like whenever this topic comes up, people go back and forth over whether or not AWS should allow you to set a hard limit and what that entails on both sides / why it’s impossible.

I feel like a better approach would be to specify what services you actually want to run and that requires a separate secret key that you will never use for other development tasks, and you have the ability set hard limits for those services.

AWS already has hard limits on things like the number of S3 buckets you can create, that require manually requesting an increase if you want more. Why can’t people say ‘I only want a maximum of 5 EC2 servers, and I don’t want the ability to spin up anything above a t3.large(translated to maybe ~20cents/hour per server)’. A similar approach can be taken for other resources.

This gets around the issue of ‘what should AWS do if you hit your spending cap’ by allowing you to set an upper bound in the amount of hardware you can actually spin up. It also solves the issue of a compromised dev key or root user account that would likely trivially allow someone to remove a spending limit anyway.

> I feel like a better approach would be to specify what services you actually want to run and that requires a separate secret key that you will never use for other development tasks,

You can get pretty far along by never using the root user, and having very limited iam policy for your day-to-day user. That is also what AWS documents as best-practice:

> We strongly recommend that you do not use the root user for your everyday tasks, even the administrative ones.

https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practi...

You can not set quotas that way, but you can at least limit which services are accessible. Furthermore e.g. ec2 has fairly rich set of conditions you can use in policies to lock down the access even more:

https://docs.aws.amazon.com/service-authorization/latest/ref...

This is the right solution. Secure your root account, don't share the credentials, use 2FA, etc. Make role-based accounts with limited permissions and never put anything in the root account. AWS recommends it, and their UI encourages it, but doesn't require it. Third-party auditing services like Vanta are great for identifying issues like this that should be addressed.

That said, AWS should make it easier or even require it. AWS is full of footguns and weak security defaults and small players (like the person who tweeted about the bill) lack the knowledge and experience to adequately defend themselves.

I know not to use the root user and am aware of many of the advanced policy conditions you can use to help control usage, but that doesn’t address the ability for service usage to quickly spiral out of control from either simple mistakes or a compromised user. People make mistakes. People’s accounts get compromised. I’m sure AWS has plenty of data on what services are most likely to be abused and could come up with a UX that minimizes or prevents that abuse in a simpler way than just providing hundreds of configuration options and calling it a day.

Happened to me, just a bit over $50k. Took a couple hours of work on my part (they asked me to enable CloudTrail and do some additional securing), and they removed the charges.

Also, it's less of a "hack" than a likely publishing of AWS keys (if you publish them in a public Github repo, they're compromised in seconds)

I took his advice and went to the Cost Anomaly Detection console and started to set up an alert for my $0.56/mo S3 charges. I want individual alerts, so I have to use SNS. I want a text message for an alert, so I am pushed into creating a 10DLC origin (even while in the SMS sandbox). A 10DLC registration is $50 for a brand or company, and $50 for each "campaign", plus $15/mo for each campaign. Am I missing something?

I haven't used my old AWS account in a while, it's just sitting dormant. Since it's old it's linked to my Amazon account (root credentials == Amazon.com credentials).

I don't want to monitor it and these stories give me some anxiety.

I guess I'm supposed to close my AWS account? I wonder if this is even possible if it's the same credentials as your Amazon account.

This happened to me. Luckily it was caught before the bill went over $500, and Amazon refunded all the charges. I have no idea how my account was compromised, and Amazon wasn't able to help me investigate. And they made me go through and delete resources across several different regions in a painstaking manual process that took me an entire work day.

They stole is ROOT credentials.

This is not about sneaky AWS, this is about a hack. There is not much you can do if someone gets root access to your system.

Setting up a billing alert to notify you your aggregate cost exceeds a certain limit takes about 90 seconds. I don't understand the overwhelming friction to the AWS billing tools in this thread.

First, the complainant needs to take ownership of poor secrets generation and management. Their account wouldn't have been hacked if they used strong password policies.

There is also the remote possibility they're playing the victim to get out of charges after they realized it wasn't economical to mine on AWS.

Annoyingly, AWS won't allow you to lower the quotas aka service limits. They can only go up. Even after you completed the project and don't need them anymore, they remain forever and create an increased risk.

I've had two machines on Google Cloud that racked up $500 in charges from cryptocurrency thieves. They send me alerts but it is so hard to get through the billing dashboards and understand what is legitimate.

After seeing this I logged into my existing but not really used AWS account and enabled MFA. I did not get any email alert or notification about to change which seems very weird

Can you use a Visa gift card with a $200 amount? Then you could just use a fake name, fake email, done. Need more money? - just add a new gift card.

Just curious, is there any other way to monetize compute/memory/space power apart from cryptocurrency?

Make software products or services that people want to buy, and sell them.

This guy is toast, without evidence to prove he has no relation to this hacker, the charges will not reverse.

Worst part is that the cryptominer probably made $1.32...

Honestly AWS is just a casino; people are blowing ridiculous amounts on RNG to try and find nice hash tables that could potentially make money by manipulating cattle.

Surely you could think of at least one more use case for AWS.

Not one that requires me to be on the hook for potentially infinite money.

I get that some alerts should be sent, that AWS should scan for common scams, react faster to support tickets etc. But how does one get enough access to someone's AWS account to create a lambda that mines crypto? Something has an access to your credit card and bills you monthly, yet you do not have 2FA? I would understand if this was some dependency chain attack, but this is just a new lambda with an .sh script. Also what AWS income has to do to with his issue?

From a later tweet:

"I don't know how my key leaked. It's a 9yr old account, under utilized + running old projects.

AWS do have alerting systems you can use to catch overspend. You should.

They didn't exist when I signed up. And with 200 links in their main menu, they're not easy to find."

Don't blame the user for what is essentially Amazon's problem. They don't have a way to set a hard limit on spend. They don't even do basic velocity tracking apparently.

I'm all up for hating corporations but how is this Amazon's problem/fault? You leaked a key, what do you expect Amazon to do? This is like software development 101, protect any kind of secret/key, don't store/send it in plain text, don't make it public. He's using arguments like 'but as a solo founder...', what kind of excuse is this? I fully agree that AWS could improve detection of these kinds of things and that the guy needs actual support but this whole twitter thread sounds like he is blaming AWS for his own mistake. Just ask for help, don't blame them. No one is forcing anyone to use AWS, you know what you get into when you make an account.

They expect Amazon to allow people to set maximize spending amounts on their accounts. They have steadfastly refused to implement this.

In my eyes, the situation is simple: human beings are fallible. If there's a chance of your credentials leaking and being exploited that way, that will most certainly happen to someone sooner or later.

I recall a VPS that i had years ago being hijacked due to Docker socket being exposed publicly (when i set one up for testing and was just learning about Docker), which was mining crypto throughout the night before i noticed it.

Imagine what would happen if there had been some sort of auto scaling in place and i had used a managed Kubernetes solution (many of which didn't yet exist back then)...

Is it Amazon's problem? Not really, apart from the bad PR that charging private individuals exorbitant sums (for them) might cause, so sometimes these bills are waived after people mess up. For the corporation at the size of Amazon, that's probably not too much in the grand scheme of things, but it's also understandable why people might be appropriately upset about not being able to set hard spending limits, merely alerts (which aren't enabled by default).

That's just the reality that we live in for now, however one can also think about why a corporation that cares a lot about uptime might not necessarily want to implement functionality that'd let resources be easily cut off when spend limits are hit or something similar, since most businesses that utilize their services might just swallow the occasional expensive bill like that anyways.

It would also be software dev 101 for an organisation of AWS's size and clout to do some pretty basic things - If someone's costs increase drastically (ie spending in 24hrs what they spent in the last year or etc.) then maybe flag it up to them - If the above is combined with costs on services/regions they've not used, definitely maybe contact them? - If the above is combined with a well known exploit used on compromised accounts, again maybe contact them?

Not saying the OP has no responsibility here. Just that I don't think it is that black and white. There is plenty AWS could do to avoid this sort of thing happening. Totally fair to call them out for that. Especially when the cost of phoning support is $2-3k specifically because of the new costs on your account!

Blaming the victim for AWS having inadequate spending controls is kinda crummy.

Absolute spending caps, or spending growth rate limits, would both be useful even outside of situations where a user's credential is compromised.

One example: When teaching colleagues how to use AWS tooling, I gave them full access to a sandbox account where they can learn without disrupting anybody else's infra. It would be nice to be able to place a spending cap on that account.

Another example: With things like autoscaling, it's very easy to accidentally misconfigure it to where it will happily start launching tons and tons of VMs. You might do this and not notice. If I had the ability to voluntarily opt into limits on how fast I can increase the aggregate $/second spend rate, then AWS could alert me in general when I'm doing something that exceeds that soft limit.

Trying to turn this into him somehow being victimized by AWS and the original comment as being victim blaming in this regard is kind of crummy.

The OP was a victim of the hacker and not AWS.

I did not say that this person was victimized by AWS, you are putting words in my mouth. We agree that this person has been victimized by a criminal.

My point is that AWS failing to implement spending controls makes this type of attack significantly more damaging, and the slow turnaround on CloudWatch spending alarms means five or six-figure sums of money can be spent before customers are alerted.

AWS can and should do more to give its customers additional control over spending as a defense-in-depth measure. This would be valuable both to limit unauthorized spending (this scenario, or a "rogue employee" scenario) and also unanticipated spending ("autoscaling gone wild" type scenarios).

AWS is a big target, and the fact that credentials can be used to spend an unlimited quantity of money means they are a huge target for criminals. Defense in depth would be wise in this situation!

I expect them to allow me to set a limit on monthly spend, and cut my access to limit further costs when that is exceeded.

Security needs to have depth. The one thing we have learned in these few decades of learning how to do online security is that any single measure can fail, and given enough scale, will fail. For anything to work, you need to be able to layer your security measures so that one failure, no matter how grave, will be fatal. AWS could add a very simple security measure here, making their service dramatically safer for their users, but they choose not to.

If I leave my credit card somewhere or the details gets stolen some how, so a thief gets it I do not have unlimited liability for the abuse of the card.

I'm very sorry Ms. or Mrs. msh.

But the 3.4 trillion dollars have to be paid back by Monday! And take better care of your information next time. It is the duty of every customer to check cash dispenser for skimmers before using them! This is clearly written out in out general terms and conditions!

I wonder if this is something Amazon just doesn’t want to solve? Maybe they did a cost-benefit analysis between making AWS extremely quick to get started vs adding a bunch of security and intelligent lockout features, etc and found that the problem is not big enough to justify degrading the user experience?

Every time I see a story like this (and there have been many), Amazon always ends up dropping the bill. Idk what this one hack on Lambda actually cost Amazon, but I’m guessing it’s nowhere near $45k.

>Maybe they did a cost-benefit analysis between making AWS extremely quick to get started

Yeah, one key issue is that it's deceptively easy to set something up in AWS, but there's actually a ton of customizability, complexity and potential gotchas lurking in the background.

They've essentially SaaS-ified sysadmin, network admin, DBA, etc. roles. So relatively casual users are pointing, clicking, copying and pasting like they're just navigating any ol' app, but beneath the surface they're commanding potentially massive/unlimited resources with the attendant costs those resources impute.

Makes you wonder whether it's just too much power to casually have on offer.

Reply on news.ycombinator.com