Hacker News Reader

Comment by markus_zhang | Hacker News Reader

We are a medium large company (close to 10k employees globally) and the main focus of this year of the data teams is to figure out every possibility to reduce Google Cloud costs.

The savings could be in 7 digits TBH annually. There are so many low hanging fruits no one cares when the day is good. Tables that have no partition expiration that grow into hundreds or even thousands of TBs, buckets with same amount of data that no one looks at, VMs that no service uses, large clusters well above their load (e.g. only 20% used maximum -- and a full replica too), very bad ETL pipelines that cost hundreds of bucks EVERY RUN, very costly queries generated by Looker dashboards EVERY view...there are just too many.

View on HN

I think FAANG centric folks underestimate how many fairly big orgs are in the same situation.

Big top down push to move to public cloud, basically no thought to operating expenses.. and now theres a ton of low hanging fruit. Most of these same firms have very little ability to measure their costs in a timely, granular manner, and so put in only very blunt processes to try to limit costs.

Last org I was at could probably re-allocate 10% of engineering staff using the cloud to a SWAT team finding cloud cost saves for 2 years before they ran out of stuff that costs 5x their salary to get fix/optimize/kill in the cloud.

A lot of the same orgs CTOs sold themselves on the idea that they'd need less infra people in the cloud. This really has not been the case, and more often the cloud infra people are more expensive too. Meanwhile many need to permanently maintain an on-prem footprint for latency/legal/legacy/etc reasons.. so you end up with net more staff.

I know a handful of Wall St firms that are moving a lot of "big data" type use cases back out of the cloud because its hideously expensive for their use cases.

I wonder if we will also see a dropoff in attendance for big tech conferences like reInvent this year.

Given how many devs were needlessly getting sent at the shops I worked (think 20% of dev team), yes...

I have been saying this for years - hybrid cloud has always been the way to go. Put your highly variable workloads in the cloud and autoscale. And anything super GPU or compute hungry that are regular workloads should be run yourself.

Bonus points if you can autoscale those same on prem workloads into the cloud provider of your choice seamlessly

Yeah also sometimes physical hardware is a constraint on waste. Quality & performance of code often has a way of degrading to the point it saturates the available hardware.

In the absence of friction to buy more immediate incremental capacity, teams may never optimize their code..

This may be good when you are "moving fast & breaking things".

This may be bad when you are trying to operate an already-scaled business profitably.

I think that is probably the right way to go. We have a put a few things on AWS which really don't fit there too well, all on the benefits of shutting down our on-prem estate, 5 years later we still have an on-prem estate.

Ah yes. Ours was to be a 2 year migration tour. Then after 2 years, they said 5 more years. I guess we will see what it looks like in 2030. I can tell you 99% of usage is still on-prem today.

I’m sure Gilligan’s Island In The Cloud would have an audience.

It's the way to go but the battle you have to fight is maintaining internal competency to host the infra. Especially if you have any kind of security or regulatory overlay its going to mean almost full duplication of a giant stack of processes that parallel each other in cloud / on prem. I think the key as you have outlined is to break out the simple batch compute and CPU/memory intensive or data volume heavy tasks as these are what generate the cloud bills. In general these aren't the focus of your policy constraints, especially if you can claim the data is not "at rest" within these systems.

It pains me to tell you this, but "simple batch compute and CPU/memory intensive or data volume heavy tasks " were the first ones my firm has, unsuccessfully, tried to move to the cloud, no joke.

From my experience I think maybe internal reporting fits self hosting. You don't need to scale too much, it can tolerate some fails, etc. But I guess it's still complicated when you throw a wrench called BI into the machinery.

The thing is that self hosting kind of already has cost gates. Whenever I needed to buy another $100K of servers, we had to make a deck and have a skip level meeting 2-3 levels up to explain the capacity planning spend.

On the other hand, in AWS, at my last shop.. we were just .. doing stuff. We at one point noticed a non-production RDS instance was going to run us $60k/year.

This was for some small part of our processing, that in on-prem land would just occupy a slice of one of our $30k servers. But in cloud, we had the pleasure of $5k/month until we noticed & turned it off.

At the scale our estate worked, we had easily 100 entities that could trigger this kind of spend.

well, its getting into the rearview mirror now, but spectre/meltdown-type vulnerabilities are a big motivator for leaving on-prem. even if you later leave the cloud for on-prem, the period without any phys infra means you can start on a clean slate with better equipment + better DC rental contracts.

oh and it really sucks when your brand new SSD's arrive at your main DC, after a 4 month wait, you open the box and the boxes are empty - someone robbed ya. Those expenses are only linked to the infra team, but the effects reverberate through the engineering org.

We did the same thing (very small company, but outsized cloud hosting costs due to the nature of our business), and frankly the biggest cost savings for us was shifting some resources off AWS and to other providers.

For us, Cloudfront was a massive expense that is without exagerration an order of magnitude more expensive than other providers, and even in the world of AWS custom pricing, it's a struggle to get cloudfront costs down to even the retail offering of other providers.

FinOps is definitely becoming a pretty hot topic. And, yeah, a lot of companies are realizing that they have some pretty low hanging fruit where they can cut spend significantly.

Feels like there is a startup in there? Give it your account number, and read only access keys, and it tells you what and where and how to change things.

There are companies out there. I expect to see a number of them when I'm at Kubecon in a month or so.

Just a note that Google itself is doing that so one has to better than that. I think the ideal candidate is a senior data architect/ops person who has xtensive hand on experience. Sadly I'm so far from it.

Sigh, I'm late for everything in my life.

FWIW the features that GCP exposes for this is way better/easier than AWS's equivalent. So much better

I never used AWS (and am just a peasant using GCP so don't know everything). What differences did you find to be extremely helpful?

GCP puts automatic analysis recommendations in the UI right next to the resources in the table of VM/etc names, and IMO the recommendations (for both upsizing and downsizing) are dramatically more accurate than what AWS gives you

Disparaging yourself is a good way to make sure you don't achieve anything. You should stop.

Lots of players in this space. At my work we have:

https://www.apptio.com/products/cloudability/

Just curious what do you think about the service? Is it a lot deeper than what Google itself provides? We are just using Google own service to turn down costs.

So Apptio is a larger application for finance people, with some cloud cost management built in. They are the main users, I just have read only access an login perhaps monthly to view budgets and cloud spend.

I just have access to small bits and pieces, but what I have seen I like.

Thanks for sharing!

Yeah. I can think about consultants making $$ giving full analysis of cloud costs and ways to reduce them. It's actually not very technically challenging TBH. All those low hanging fruits can be grabbed by a junior enginner or even a non tech senior person who is a deep user of such services.

I've wondered how much of the cloud spend in the world is just junk like this.

I don't mean that cynically or sarcastically. The cloud makes it legitimately more easy to deploy resources; I've experienced this in my own teams. But the flip side is that it is easier to deploy things without thinking about the costs, or deploy things and forget about them, and management can just shrug and say "yes, it's cloud, it must be cheaper and better".

This is by no means a cloud-unique problem. Watch any datacenter a company has been at for a few years get shut down and moved to the cloud. The stream of "hey, who owns this?" and "are we still using this?" and "I just yanked the network cable, has anyone noticed?" is morbidly amusing. Even if you think you're tracking everything, you quite likely aren't, because the owners don't update their own tracking info very well.

But cloud makes this worse.

So I just find myself wondering how much of AWS profit is on this sort of stuff.

I myself have S3 buckets full of crap. As I do watch my spend, said buckets full of crap are in the ~10-50$/month range. At that price point it is difficult to justify engineering time to clean out the crap. But it's not hard to imagine that scaling up by three or more factors of magnitude.

Yeah exactly. I feel that recklessly opening new projects and then abandoned them, or refactored multiple times with different teams, quickly added up to the number of rss no one owns.

Last couple of weeks the whole data org underwent a forced upgrade of certain components on Google cloud and we got quite of clusters no one knows who owns.

I don't know but I'm itchy to start finding ways to build new projects because that is the best way for techy people to grab ownership and promition.

We saw this one iteration back with VMWare. Once it became easy to spin up a VM without getting a PO for a server and finding rack space, the number of "servers" in the org exploded. And of course we lost track of what they all did, and who they belonged to... but there was actually a healthy ecosystem of tools built for (and by) VMWare to manage this situation. The cloud providers seem incentivised to make tracking real costs as difficult as possible. At least with VMWare once you paid for it, it was yours.

There’s great tooling on cloud providers for tracking costs. The pain point is that if you never set it up it’s incredibly difficult and time consuming to add it after the fact through tagging.

Start by owning the datacenter. You might just as well have the data that no one looks at and the 20% load clusters in one, two, or a few 48U racks [1] in the room next to you. Keep whatever is latency-sensitive still on someone else's computer, i.e. the cloud.

[1] Get a "server as they should be", https://oxide.computer (no affiliation)

> Start by owning the datacenter

But then, when uncontrolled inefficiencies need to gobble up more hardware, you would have to buy, install, network, maintain, and understand this new hardware.

Unused resources do not gobble up real hardware.

Real inefficiencies do, but then you will have the more detailed data the GP is talking about.

I really want to learn in an on prem env (and learning on my own 32GB 2TB x-handed server is probably just scratching the surface). Sadly I'm a junior so could not make that call. I also have no idea how much cost the transfer to on prem will be.

No time like today.

You don't need to make a large datacenter, or to migrate everything at once. Find some low-risk services, get some consultant help you improve some room on your premises (you'll need to care about security, electricity and cooling), get a pair of network contracts and one hack of servers (or maybe even 4 servers), and move those services. Favor things that are used internally.

If you are looking for 7 figures of savings just by picking the easy fruit, you will almost certainly get positive ROI on that. Just do not move critical services if you don't know what you are doing.

I work for a tiny startup and while things are good, we are prioritizing reducing cloud costs as well. Our bill is relatively tiny right now, but as we grow it’ll become a bit more unsightly. Best to do it now and get that over with!

This, and cloud security, have become my niche at my small company. Keeping these things tidy now will save us so much headache later. It's a bit of a battle fighting product for time but these are hills worth dying on imo.

Interesting. I wonder how one learns it. I guess I first need to jump to a full Cloud admin position to learn on job.

I don't really have a secret. I just have 8+ years experience with one cloud provider across 4 different companies as an SDE, so I've seen all the highs and lows that come with using it. Current company keeps bugging me to take some certs but doesn't seem worth the effort..

Thanks, I guess I just need more experience. But first let me move to a real cloud admin position...

At a tiny startup I'd be more concerned about selling product/delivering to get the next round and keep the company running than saving few 100$ on infra cost.

Not every startup is funded by VC vultures that set aggressive timelines

Deadlines don't matter - if you're in a startup you're trying to test the market fit and iterateing to corner the market. Worrying about micro optimizing infra cost and talking about scaling in this context in my opinion is premature optimization - root of all evil.

Yep, we're in the same boat. About 4mo ago we started looking more closely at AWS costs. Tons of low-hanging fruit.

Yep I’m in the same size company and we’re doing work to reduce GCP costs. You are so right, our team did a sprint or two of work recently and shaved $100k off our monthly bill which in our company is small potatoes

just messy bread making machines, as joel spoelsky would likely put it

https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...

Reply on news.ycombinator.com