Hacker News Reader

Comment by jerf | Hacker News Reader

So, I have a personal AWS account, and setting up cost controls seems like a good idea. Let's try it out.

"Welcome to AWS Cost Management

Since this is your first visit, it will take some time to prepare your cost and usage data. Please check back in 24 hours."

My "cost and usage data" is that I have an S3 account costing ~$10 a month containing my cloud backups of my family's high-priority data, for maybe the last 18 months. That's it. I just reloaded in case 24 hours was some sort of pessimistic estimate and they actually had it done in a couple of minutes, but the message persists.

WTF.

I can set up any of dozens of services in thirty seconds and spend thousands, but it's going to take 24 hours to understand ~$100 in costs over the course of the last year? Someone could read aloud the entire history of the account, every S3 transaction I did, in that amount of time.

I can really feel the love here.

View on HN

AWS doesn't benefit by providing excellent cost controls and visibility into spending.

I'm not saying that neglecting this is a conscious, cynical choice. It may just be that there's simple no incentive to build it in the first place. A self-motivated engineer or team manager is probably not going to say, "I know! Instead of working on the next bit of shiny exciting tech, I'm going to work on cost controls!" Though sure, it could also be deliberate.

That's not true at all. Businesses want to reduce risk. Providing cost controls assuages a client's fears. If you're picking between two clouds, cost is a concern, and cost risk is a big factor in that. Would you put your career on the line for a riskier cloud? That's not only cost, but also security, privacy, legal risk, compliance, etc. Clouds spend a great deal of resources to improve all these things, especially for highly-regulated industries which constitute a large part of the potential customer-base: health care, banking, government, polluters.

And yet AWS has awful cost controls. My interpretation of why they have bad tools in this area might be wrong, but what is the alternative explanation? Incompetence? Certainly they must know it's a concern for a subset of their users.

You also may not be cynical enough here: Lookup the Amex tax avoidance strategy that they pitched to companies for a while before a whistle blower tipped off the IRS [0]. Or pretty much any fraud that originates at a high level in a corporation. Companies make risk/reward decisions about doing bad things all of the time. ::ahem Theranos::

[0] https://www.wsj.com/articles/amex-pitched-business-customers...

From a few different devs at AWS I have heard a couple of common explanations:

- Billing code/codebase and everything around it is (was?) a big crapshow and people are naturally reluctant to stick their hands in something so critical and hard to adjust (and equally reluctant to approve such projects).

- Some teams believe that customers don't want such a feature and may have been explicitly told that they don't want that feature to exist. The latter part seemed like hyperbole to me but I don't have the context they have.

naturally reluctant to stick their hands in something so critical and hard to adjust

That pretty much lines up with my comment about engineers that would prefer to work on some bit of shiny new tech that's more glamorous than cost controls, and my general sentiment that AWS simply doesn't benefit from pursuing these changes.

I imagine it's also the fact that you are rewarded for working on complicated projects that impact the bottom line positively.

Driving this billing project is going to be thankless work, and at the end of it Amazon will lose money.

It's a lose-lose.

> - Some teams believe that customers don't want such a feature and may have been explicitly told that they don't want that feature to exist. The latter part seemed like hyperbole to me but I don't have the context they have.

I have heard such reasoning many times to get away with all sort of BS.

The argument is often impossible to refute, since you don't have any customer at the meeting and the feature is on the idea stage. So I guess it is some sort of argumentation fallacy, referring to a non-existing person?

So I guess it is some sort of argumentation fallacy, referring to a non-existing person?

The person might exist, but either way I'd say the fallacy is an improper argument from authority, in two stages: First, AWS is claiming authority on the matter and citing other customers as their evidence. That evidence is an improper appeal to authority as well because the expertise (or even existence) of the person(s) behind it cannot be validated. Sort of like a celebrity endorsement where you don't even know who the celebrity is-- e.g., An unnamed person who endorses a basketball sneaker but you don't even know who if they play professionally and actually use the sneakers themselves.

For that matter, even a proper appeal to authority (area of expertise is relevant to the topic at hand, expertise had been validated, there are no conflicts of interest like getting paid for it) are not very strong evidence for a claim. They are indirect evidence where the person is saying "You don't need the data, I've reviewed it and you can trust me".

The likely explanation is that there hasn't been enough pressure put on them to improve these things. They care primarily about large customers, and they already have plenty of ways to resolve cost issues, and usually sign custom contracts. If GCP or Azure has a significantly better answer to this problem, AWS might start caring, but I don't think they do. (I work at GCP.)

I think that lines up nicely with my general point that AWS simply doesn't benefit much from putting resources into this area, although some C-level folks might be enticed by shiny marketing materials about "Best in Class Cost Controls & Monitoring"

For other option, it does seem like GCP (I don't know much about Azure in this) is a little better in this area. Admittedly, I'm speaking from limited experience playing around with free tiers in both, along with the stream of stories that we hear about AWS.

I think it's also fair to point out that cost controls may not have helped much in the example of this particular article: If your account gets hacked, presumably the hacker will also have some access to remove cost controls.

Realistically, this wont be a priority until Gartner has a category for this and places all these cloud players on a ranking scale. That sort of thing seems to really move the needle for execs.

Ah, yes. I always give a little internal eye roll when a vendor highlights their placement in a magic quadrant.

If you are a small team and worried about cost, which everyone usual is, the best bet is not to use AWS at all and go with something like DigitalOcean or Hetzner where the pricing tends to come with less surprises.

And setting up two-factor authentication is far easier.

That seems reasonable, although the argument for AWS is usually that it can scale much more easily. If your primary use case is something like EC2, that's probably a non issue when choosing DO unless you have very specific needs for your instances.

> AWS doesn't benefit by providing excellent cost controls and visibility into spending.

Except you can create a budget that is based on individual services or usage instances. And you can view each service's cost in zoom-able billing statement.

What information are you unable to find about your bill?

(NOTE: I'm not an AWS shill. Every time I see AWS billing threads I'm concerned I'm missing something obvious and putting myself and my biz at risk, so I drill down on posts to make sure I know what I'm doing.)

Not to mention AWS only updates billing data 2-4 times per day. So even if you do have this enabled, on an accrued cost basis you're going to have to wait 6-8 hours to know about it from Cost Explorer.

Many changes to Google Workspace take up to 24 hours, as well. It always blows me away when I make a tiny change and get that message.

The billing system runs each night.

This.

This is absurd. I just logged into the billing alerts console. I set up a total billing threshold in 90 seconds:

https://console.aws.amazon.com/billing/home?#/budgets/overvi...

What on earth are you getting at?

You can setup whatever alert threshold you want, but it doesn't stop charges from accruing.

What is needed is a way to say "I only expect charges of $10 or less per month. If charges go over that, disable the account until I can get in there and find out what is happening."

That can also be done. I can shut down any specific or cluster of EC2 instances if any one exceeds a $ limit, or all of them. The resolution is up to me and I immediately stop getting charged.

If I have lost ROOT access, as the OP did, nothing can save me: all alerts and thresholds can be shut off by the hacker. Just like any other system.

I saw that part about setting up quotas and actions when budgets are exceeded. I did setup an email alert, but the quota thing was extremely complex. There was a list of service quotas a mile long just under the EC2 section. Some of them were automatically set by AWS, most of them were zero. This is clearly aimed at companies with a huge AWS infrastructure investment and the staff to manage it, not some schmuck individual like myself that just wants to not get a $45K bill.

Again, the $45K was due to a hack of root credentials. Not AWS being shady. If anyone got access to root on your machine you'd be screwed.

If a total shutdown when aggregate monthly budget is triggered is insufficient, and fine-grained quotas are too complex, you probably shouldn't be in the cloud because Azure and Google are equally complex. Or do you like another cloud-computer service better than AWS because their budgeting quota systems are better? Please share, I'd like to know what you use.

> If a total shutdown when aggregate monthly budget is triggered is insufficient

I never said that. In fact, that's what I'd prefer. I don't use AWS for anything critical, only for testing, so I'm fine with my account getting temporarily disabled or all requests returning error codes.

> Please share, I'd like to know what you use.

B2. B2 has a 10GB free tier for object storage and returns an error code if that limit is exceeded without a credit card on file. They have limits (called Caps), like AWS, but B2 apparently automatically sets their limits low, because when I checked, my limits are $1, $5, $1, $1. Pretty sure I never set these. They will email or text when limits are exceeded by 70% and 100%. As far as I can tell, they don't have a hard limit (return error codes) once a credit card is on file like they do with no credit card on file, but I think that would be a good thing to add.

Have you ever visited the page before? The message explicitly says that because this is my first time they need 24 hours to set it up.

I can only report what happened. Why this couldn't be preloaded for my teeny tiny account and its probably on the order of dozens of kilobytes of data is something you'd have to take up with Amazon, not me.

Yeah good luck with anything other than e-mail though, if you want SMS notifications you have to go through setting them up through a layer of indirection through AWS SNS messaging.

And it'd be nice to just have a $100/month killswitch on my personal AWS account and I have no idea how possible that is with their actions (which seems to require configuring IAM accounts). I don't run anything "production" there and I'd be happy to have all my shit suspended if that dollar figure was exceeded, but fuck if I know how to do that without taking a week to catch up on the past decade of how to manage infrastructure with AWS.

It may be simpler to just suspend my AWS account entirely. I think I only have some ancientsauce mysql backups on S3 which are costing me a few pennies a month.

www.privacy.com

Reply on news.ycombinator.com