Hacker News Reader

Comment by pram | Hacker News Reader

Definitely not. My company has been drastically scaling back our AWS spend and I mean about a $10m+ a month reduction. I guarantee we're not alone. All the gee-whiz resume-driven-development type project infrastructure was the first to go lol.

View on HN

It's easy to accidentally waste a lot of money on AWS. My company cut our costs from ~20k/month to ~6.5k/month with no degradation of UX or dev velocity. It was all due to unnecessary redundant builds and integration tests and lingering unused resources. We used the CloudForecast service to identify and track costs (https://cloudforecast.io/). An underrated side effect of this service is that making costs visible to the engineers incurring the costs can encourage them to spend less.

Pretty sure a typo and the million wanted to be a kilo?

$120m/year for apparently unnecessary IT capacity would be hard to hide even in government contracts...

Its AWS. Isn't "obscure high costs" the name of the game? But now people have the time and incentive to dig into those costs more critically.

I guess the question is why would you wait to make savings like that? $10m/month would pay for a small 'cloud cost cutting' team with plenty of change left over.

$120m is not ridiculous for a medium to large enterprise.

For an engineering team of ~1000, just the build servers will cost about $1-5m a month (depending on what's being done there).

$1000+/month/per engineer sounds way too high.

seriously, that's a very beefy server per engineer, just in cloud resources. not including what they use to do their day to day work. NFW would I allocate such a spends per engineer--maybe for beefy projects/groups, but not for the org as a whole.

Depends on how many engineers you have.

It’s not just costs incurred by engineers, it’s costs incurred by users. If you’re providing file storage as part of your app, then for every engineer with a dev environment in the cloud, 1000 users could be uploading gigabyte files into your S3 bucket.

That seems... excessive.

Let's say we pair engineers up just to make my math a little easier here. One engineer's allotment goes to compute resources, another goes to storage.

At the low end, that's a 32 core/128GB RAM build server with 10TB of storage per pair.

At the high end, depending which way you spec that out, that's 96 core/385GB RAM with 50TB per pair.

What the hell are you guys building?!

I work for a company with ~1000 engineers, and our build server spend is much lower than what you suggest. If we were spending even $10k/mo on build servers, heads would roll.

$1M/mo will rent you about 300 c5d.24xlarge servers, at on-demand rates (and of course no medium or large enterprise is paying the higher on-demand rates). I'm pretty sure we're running somewhere on the order of 10 build servers, and they're maybe 4xlarge or 8xlarge boxes.

I’ve seen a big bank spin up a dev environment in a cloud for a lending platform at a cost north of $36M a year before they finished the requirements for the project.

Not impossible at all to believe they could cut $10M/month from could spend at a big enough company just by trying.

I know of a few companies burning AWS at around that rate, though they are outliers. And yes, 90+% of that is lighting money on fire. In the list of business priorities, cost containment is pretty low for some companies when capital is cheap. There is a much larger cohort of companies spending much less but wasting just as much.

I once saw a major west coast health insurer purchase $50MM+ in hardware to solve what could have been at most a couple million software problem. In fact, had they negotiated well they might have been able to pawn the whole problem off on their vendor.

And here I was pretty proud of myself for reducing our monthly bill from $5K a month to $2K… we have some basic sports data APIs that are hardly being used since there's no live sport being played, so I've removed most of our redundancy for the time being.

A $10m/month reduction is quite significant. Question is, was your company spending $10m/month extra unnecessarily or have they had to cut down and go without important things in getting this kind of reduction?

A reduction in customer numbers means a reduction in the resources required.

Many businesses associated with outdoor activities, sports, travel etc etc will be seeing less traffic at the moment. Many business are pretty much closed. This will then severely impact other businesses which sell to those businesses, and so on, all the way up the chain to advertising and cloud providers.

The Ubers and Lyft's save by committing to a certain level of spend. They have paid -- might as well use the resources, anyways -- unless they've since negotiated with Amazon.

We have had to scale up our AWS spend! And any gee-whiz solutions to improving efficiency and scalability are very welcome right now.

For those wondering as well, at 10b quarterly revenue, 10m monthly is ~.3% of all AWS revenue.

It seems likely to me that a recession or pandemic affects more than just the OP's business, so while that one customer is probably meaningless to AWS, the thought process that led to scaling back spending could be meaningful.

Interestingly, I had the opposite takeaway.

0.3% of monthly revenue from a single customer's reduced spending only a couple months in to a multi-year recession is pretty alarming.

Makes me wonder if OP's company is particularly exposed to retail, travel, or another sector severely impacted by the first order effects of the virus.

That sounds like a lot to me for just one customer. But in the absence of much more data about AWS customers and revenue, it's hard to draw any meaningful conclusions.

"Cloud Scale" always promotes that you can scale up easily, but that also means you can scale down easily

Well, that's the marketing story, at least.

In my reality, a large part of my programmers' effort is spent battling various "capacity exceeded" AWS errors.

At the same time, many entertainment companies are scaling like crazy, spending magnitudes more on clouds.

My team reprioritized efficiency over features during the recession, trimming a large amount of unnecessary AWS spend that wasn’t as pressing when the economy was performing well. That specific spend will never come back to AWS, because it was always unnecessary.

I’m sure new unnecessary spending will creep in as time goes on.

having a hard time to see how you scale down 120 mil a year. How much you spend yearly now

It’s probably not for the entire year, a lot of it is just various clusters being scaled back. That said some of the stuff has been moved to colo hardware, and some of it is probably just gone forever.

The guy is lying, obviously.

Maybe, maybe not. But I think I agree with you- 10m/m is a comically large number(I wonder how many companies spend even 1/100th of that a month on cloud compute?) and I have no reason to believe this person so why would I?

Correct there is no reason to believe me. However for context and perspective our quarterly revenue is usually around $3 billion. $10 million is not comically large. I’m not talking about a startup. hth

I'm shocked that some middle manager with half a technical head on their shoulders didn't use this as a VP play before this; surely, "I saved the company 120m a year with no downstream consequences to yada yada" would have been worth it for someone somewhere to climb the ladder.

Why would you, if in good times you can enlarge your own budget/headcount/importance/... just by increasing budget no questions asked, that's a safer way ahead. You risk devaluing your own position in the company for an uncertain gain...

I was obviously being slightly sarcastic. I wouldn't want to personally work that way. I do believe that might be quite an effective strategy if one's goal were climbint the corporate ladder.

I think it would be easy to hit especially if you were spending on cloud SAAS offerings + actually have a lot of users. I know I've managed to spend a few $10K/month on a project before - and it was actually celebrated because it was less than half the cost of what it was replacing.

Yeah but that’s almost my point; 20k/m is an order of magnitude less than 10 million/month.

Technically, 2.5 orders of magnitude.

Thank you. I need to get better at using orders of magnitude.

Apple spends more than $30M/mo. on AWS. [1]

Comcast, I was told, spends over $1B/year on AWS but I cannot find any public references to back it up.

[1] https://www.cnbc.com/2019/04/22/apple-spends-more-than-30-mi...

If you had to do it all over again, would you still choose AWS? Or would you go with another cloud, or perhaps even non-managed bare metal?

Reply on news.ycombinator.com