Hacker News Reader

Comment by onion2k | Hacker News Reader

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1), the price of building your own services, the price of software you need to buy, and so on.

You only save money if the differential cost based on all the factors is lower. AWS is expensive for some things so it often does save some money, but not always, and if you haven't done a proper analysis you can't know.

View on HN

>> moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops

This is the traditional “sell” for cloud computing and I don’t buy it at all.

The clouds would have you believe it doesn't make sense to run your own systems because you need too much specialist expertise to run your own systems.

The clouds sales pitch is the if you go cloud then you don’t need all these specialists.

That’s rubbish. Cloud operations need the same or more headcount if technical specialists, they’re just doing different things.

The old “don’t run your own systems, it’s cheaper and easier to go cloud” is just sales fiction.

Don’t believe it.

Don’t believe it.

You don't need to 'believe'. You need to do an analysis of what you need and how much each option will cost. Then you can know.

If your argument is "I believe onprem saves money" or "I bet AWS is cheaper" or "Jim Morrison came to me in a dream and said I should use Azure" then you haven't done enough research.

> you haven't done enough research.

the research isn't free either. The more indepth and time it takes to do such research, the slower you come to a decision and ship.

Cloud allows you to ship fast. It allows you to go without research - just accept the marketing, and pay up.

You pay above-cost (compared to on-prem) when you grow to a certain size. But this is usually a worthy trade off tbh.

It's a good way to bypass legacy IT teams. A onprem server will have a bunch of snake oil endpoint protection products running on the box, bespoke config changes and take a couple months to get up and running, a ECS container is up and running in minutes and has whatever you shipped in the container without all the commentary from the peanut gallery.

A poor DX can happen in the cloud as well. Waiting weeks for IAM configuration to be solved or a security group to be opened..

I've run into this kind of snake oil in the cloud, as well. Some organizations demand all routing go through an "upstream" VPC operated by the parent org's IT dept, so various third-party security services (WAFs, IDSes, etc.) can scan / inspect the traffic.

There are hundreds of factors for each company and application. Sometimes it works and sometimes it doesn’t, but it’s rarely as simple as retail prices.

Even the same number of "technical specialists" doesn't mean it's the same if one option lets you move faster or remain more reliable.

Datacenters are expensive to run, for many reasons. That is the appeal of the Cloud right there, before we even start talking about the people aspect.

I challenge any in-house team to achieve the same reliability as S3.

To be fair most businesses probably don't need it, but it is worth taking into account.

the same reliability at what scale? this is a major difference. S3 has amazing reliability but is a incredible complex system with many moving parts, which incurs cost in terms of reliability. The simpler the system, the more reliable it usually is. Most bussiness do not operate at even remotely the same scale as amazon, and running a small scale cluster of S3 like functionality wouldn't be that hard.

Amazon S3 standard storage offers the following features: Backed with the Amazon S3 Service Level Agreement. Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year.

S3 is reliable as data isn't not stored in a single place. It reduces the risk of water leakage or some vandals ripping your storage rack apart.

However, as someone put it, put your money where your mouth is. Become a contractor offering to reduce storage costs in exchange for 10% of the savings. You should be a millionaire in no time, according to your narrative.

> That’s rubbish. Cloud operations need the same or more headcount if technical specialists, they’re just doing different things.

It's a capex vs opex question. For financial quackery reasons, accountants and stock markets prefer having less people on the direct payroll, which is why everything not part of the "core business" is outsourced - even if it is more expensive in either short or long term.

When I moved reddit from on-prem to the cloud, I cut our costs by 27%. It was the same number of people managing both infrastructures, but once we moved to the cloud, I no longer spent most of my time imaging machines and driving them to a datacenter to rack and stack them. Instead I spent my time coding ways to manage machines via API, so that when I needed to double our infrastructure, I just ran a script.

Cloud makes a ton of sense for high growth companies. If your infrastructure is mostly static, that's when it makes sense to go to a datacenter. Or if you have one very specific use case, like Dropbox.

This is very unusual in my experience, except in cases where the on-prem was either massively over-provisioned or the customer was getting absolutely gouged on pricing at the DC or IP transit or whatever.

Do you recall what the big savings were in? Compute, storage, DB etc? Be really interested to understand why this case is so different to many I've read.

Funny enough I have the exact numbers handy! Keep in mind this was 2008 and we only had 150M page views a month back then.

Data Center (per month)

Servers: $6K

Cabinet (x3): $15K

Bandwidth: $2.5K

Support: N/A

Total: $23.5K

EC2 (per month)

Servers: $13K

Storage: $1.5K

Bandwidth: $1.1K

Support: $1.2K

Total: $16.8K

Interesting, thanks - I assume on EC2 you were running your own database servers? (Just based on the fact that you don't mention RDS.) This would keep costs down significantly in most cases although I assume they get transferred into staff costs if you've got people maintaining them to get the same level of support for things like backup, replication, etc that you get for "free" in RDS...?

Yes, we ran Postgres on EC2. I admined it myself both in AWS and in the datacenter. RDS didn't exist at the time. :)

At the time we and Heroku were running the two largest Postgres clusters on EC2.

This is precisely my experience and what I observe in companies that fall into the cloud trap. In addition, it is important not to overlook the significant transformation and upskilling process required, as well as the time needed to accomplish it.

> And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1)

This is an important factor to remember when evaluating could costs. If you want to survive a cloud outage, you need a multi-AZ or multi-region deployment, and that costs developer hours. And you need to deal with the potentially catastrophic cost of inter-AZ or inter-region traffic, which can be catastrophic and/or cost more developer hours to mitigate.

> If you want to survive a cloud outage, you need a multi-AZ or multi-region deployment, and that costs developer hours.

RDS databases are one click to set up in multi-AZ, and if your stack is Kubernetes based or at least EC2 autoscale capable it isn't much more work to make it multi-AZ as well.

Multi-Region deployments however, these are indeed expensive and nasty to set up.

usually you're right but the cost mentioned here is specific to BLOB data storage ... a very specific problem that can be isolated and run on external services and save the company lots of money.

^^^

The cloud vs. on-prem argument often seems to ignore the (enormous) middle-ground. Just because one portion of your architecture would do well to be run outside the cloud, doesn't mean you take out all the other parts you don't want to deal with yourself. Furthmore, "on-prem" might mean in your building, in someone else's building co-located and you rent space and control it, or in someone else's building where they deal with most hardware, etc.

That said, maybe Canva has considered a non-AWS solution and decided against it. Or maybe they've gotten certain better deals from AWS. We can't really know for sure on the outside.

My team uses AWS for _everything_. There's three reasons for this on our team (and I've researched it for our use cases.

- Consistency. Instead of saying "Oh look in AWS for this app, Azure for this one, and hetzner for this app, except it's test env is in AWS", it all just lives in AWS. It massively simplifies docs, onboarding, and reduces the amount of one-person specialised knowledge.

- Engineering Costs. Similar to above, but in terms of engineering, there's less to know and understand. Instead of needing to know how the AWS load balancer routes/connects to a VM somewhere else, and how that VM gets it's blob-storage-data from azure, we only need to understand AWS concepts.

- Vendor Lock In. Yeah, it's there. If we have a service that uses data from S3, there's egress costs from S3 to <other provider>, but not with EC2. We've consciously accepted this lock in for the time being.

Now, we're a 50 person company so YMMV, but the above tradeoffs plus an "opinionated" setup in AWS (everything on ECS, logging to Cloudwatch, RDS for DB) drastically reduced the "ops" overhead on our side after the initial setup. If I started over, I'd make the same decisions again.

> My team uses AWS for _everything_...If I started over, I'd make the same decisions again.

Okay? I never said that sticking to a single cloud provider isn't appropriate for some (or maybe even most) people. It's good that you have a setup that you believe works well for you.

But you're missing the point above. S3 is probably the easiest service to replace - there are loads of providers which use the _exact_ same protocol as S3. It's a drop in replacement. It literally uses AWS concepts, there is nothing else to learn apart from putting a different url into your application.

Very few people should really be using S3 at any serious scale is my thoughts. The cost savings are enormous (plus cloudflare for example replicates your data a lot closer to users for no extra cost, significantly improving performance). The cost savings can be absolutely enormous for very little/no additional complexity given how many providers are compatible with S3, and the fairly 'boring' nature of S3 compared to other technologies.

People really have to be afraid of running VPS or whatever if they can't spun up a min.io instance to have their own S3 without dealing with Amazon's Bullshit

How big of a production deployment have you ran on min.io in terms of data transfer / month and total storage?

Because I've done small scale and can tell you I'd run S3 in the future.

Point taken as I haven't used it in prod properly, I'll cross that bridge when I find it but Id rather keep my money and put some time into making it work. There's also cloudflare R2 now if S3 is too expensive but self-hosted is out of the question

> - Vendor Lock In. Yeah, it's there. If we have a service that uses data from S3, there's egress costs from S3 to <other provider>, but not with EC2. We've consciously accepted this lock in for the time being.

This is where I think the FCC should take action.

To the extent that this issue is a mutually agreeable arrangement between you and Amazon, it seems obnoxious but does not seem like it rises to the level where regulators should take action. But it affects third parties too: specifically, it prevents non-AWS-hosted vendors from effectively marketing their services to you. In that regard, I think the FTC should try to put a stop to this. AWS should not be permitted to effectively subsidize its and its partners’ services over outside competitors.

(And the US Government should never have accepted cloud deals with excessive egress costs. Part of the bidding process should have been a requirement for networking outside the winning provider to be priced competitively with internal networking)

or just consider different providers

and easily lower costs by 50% depending on balance storage/bandwidth

Reply on news.ycombinator.com