The cost of cloud

I don't consider this rude or blunt, but rather incomplete as I really am not sure what points are frustrating for you or what you would hope someone takes away from it -- I'm an outside observer on the cloud subject as I have seen huge debates over use of the public cloud internally at my company and also with client companies.

I've seen the billing costs of the public cloud absolutely demolish an IT org's yearly budget in a month because of unexpected cost upticks, and I've seen a reduction in total cost of ownership by reducing needed licensing/staff/building costs. I get both sides on what the public cloud can do.

I've also seen what you can do on-premises; I've worked with clients who manage 7000+ machines (mostly virtual + some physical) with a team of 4 using pretty reasonably priced on-site hardware. (pro-tip, I guess Hitachi boxes are absurdly great servers with fantastic uptime, pockmarked only by an absolutely horrendous UI to manage)

My experience from the many clients I work with is that it is less about the specific stack you settle on and more your comfort level in getting the most efficiency out of it. The deeper and more intimate you are with all levels of your infrastructure, the better you know how to eke out the most from every single $.01 you spend on it.

Which 500 instances? 500 ec2 “large” is like half of a rack… or is the point to engage opponent in unwinnable argument?

I've learned a lot from managing school networks for public schools. Thousands of users runnung on either re-purposed or gifted hardware from 8+ years ago.

In 2013 I was running a school with 500 students and 70 teachers on a 8GB RAM HP Server that was built in 2005 and had no problems other than disk speeds for network transfers.

The same setup in the cloud would have been much more expensive but then again I had/have access to unlimited Microsoft product licenses because of the MS-ACH agreement so take that with a grain of salt. They even give every public school in the country their own unlimited KMS host key.

The cost of three person months for a reasonably competent devops person is probably close to 50000$. Maybe a bit less for a cocky junior one that will make a mess and a bit more if you pay premium freelance rates for somebody less likely to botch the job. That pays for a lot of infrastructure. Not counting your own cost is a rookie mistake. And not realizing you really need 4-6 of these people to be able to get to your five nines is the second mistake (you need people on call 24x7 and when they are sick, over Christmas, etc). So, the real cost would be closer to 1M/year. Just staffing to babysit stuff you build manually ... or you pay Amazon, Google, etc. and you just worry about your own application not crashing. That's why this is so popular.

Few companies actually need that many instances. The math for the less than 10-20 instances the vast majority of companies actually need is quite brutal. A day of your time basically pays for months/years of hosting. The thing to optimize is devops time. Not hosting cost. It's by far the most expensive thing and also the most likely thing to fail on you (by leaving, by being incompetent, negligent, lazy, sick, etc.) and also the hardest thing to source when you need more of it. Good devops people are scarce.

I've dealt with plenty of companies that had no more than two or three idling t2 instances paying for multiple devops people to babysit that "infrastructure". It's stupid and wasteful. A decent devops person costs about 0.5-1 instance year (i.e. a full year of hosting 24x7) per hour for such small instances. And scaling an instance group from 2 to 500 instances is a 1 minute job if you ever need to. Unless the savings are enormous, the time they spend on minimizing the number of instances or automating their deployment will never be worth the money. It's money down the drain. You need to think in terms of a few hours for getting stuff done to make it worth the cost. Anything more is probably too expensive.

I maintained a datacenter with approx 1000 hypervisors with a very small team and took few weeks to start having production workloads. The effort to maintain hardware was quite little and it was hugely cheaper than any cloud service.

Having said that, your requirement is pretty absurd. Billions of people choose to own and maintain houses and cars and cook their own food because it's cheaper than the alternatives. Nobody expects them to be professional mechanics or cooks.

Assuming the 500 instances are something like m5.4xlarge, the on-demand cost would be $3M. You save something on reserved instances, but have other excessive costs like egress, so that should be the right order of magnitude.

So why do you limit a project that's supposed to save $1M+ to a cost of $50k or so?

It's still about cost.

(1) opportunity cost: doing your own systems administration instead of growing your business (features, marketing, etc).

(2) switching cost: once you have a working system that outgrows the free / cheap AWS tiers, and might be cheaper to run outside cloud, switching away from the cloud becomes expensive, and does not look like a good investment to many, see (1).

When people talk about the cost of cloud, there’s some assumptions we need to state:

A. Does your workload fully utilize 100% of the capacity of the resource? If not, then cloud would be cheaper. Just like if you only need office space for a few people, it’s not cost effective to buy an entire office building. If you only need server with a few gigs of RAM, it’s not cost effective to buy (own) an entire physical server.

B. If you are going to fully utilize a resource and don’t want to purchase/own it - then a service provider needs to provide that asset to you around cost and make margin from the efficiency from scale they have. Example, it’s actually more expensive for me to buy all of the ingredients to make a hamburger than to simply buy a fully prepared hamburger for McDonalds. McDonald’s is able to provide this due to their scale.

What I’ve seen is that when you’re in Group B, many people are finding that AWS/etc is way more expensive. Essentially, their scale in efficiency is not being passed down to the customer in cost savings. And the sizable cost premium is not worth the value received in return.

I’ll give a good example of where this does make sense, and that’s Hetzner or OVH. Their scale allows them to procure & host dedicated servers at a price I’d be difficult to match doing it myself. Or even if I could beat their price, it would be minimally. But folks are finding that with AWS/etc, that premium is extreme and that’s where the equation is unbalanced for folks.

I'm not even sure it's cheaper. At a previous employer we had approximately 150k of cloud expenses per annum. Bringing that in house would have eclipsed the cloud expenses on additional staffing costs alone.

Not really. As far as I can tell, the core value of AWS is letting accountant shift CAPEX to OPEX.

The technical considerations are a distant second or third place.

Not to be dishonest towards the argument that you're making, but most of the examples that you give actually seem fairly straightforward.

I do my own taxes, but maybe that's just easier in Europe and is definitely easier for individuals. That said, there's no reason why a LibreOffice spreadsheet would be an insufficient solution for handling taxes and other things like that.

I also haven't eaten out in years, the closest to that was ordering some Wolt when hanging out with my friends pre-COVID, because they wanted to try some. Apart from that, it's all just home cooked meals for me and that's pretty great. It also seems to be working out great for the folks over at https://www.reddit.com/r/mealprep/top/?t=month

At work, the company that i work for have their own building and have their own support staff as well, which seems to be working out great for them.

Furthermore, there are plenty of on prem resources that are used and despite the disadvantage of lacking self-service in many cases, there's very little difference in configuring and running software for deployments, with something like Ansible and containers. Even moreso when you have to support clients that have their own particular data centers and on prem deployments, which might differ noticeably from public cloud offerings. That's even not thinking about things like compliance in regards to what data can be stored where.

Personally, i also have a homelab with some repurposed old computers with 200 GEs and value RAM, a few HDDs and WireGuard for working around NAT and exposing my sites to the world through a pretty cheap cloud VPS or two from https://www.time4vps.com/?affid=5294 (affiliate link, to make hosting cheaper if anyone else uses them). Of course, when i need 24/7 uptime, i do use their VPSes in a hybrid cloud setup, especially since my blog getting 30k views could be a bit taxing on a residental 4G LTE modem connection.

The argument about competencies, ecosystems, 3rd party offerings, outsourcing and so on is probably a valid for some, but not for me and not for many companies out there - too often you end up depending on SaaS solutions which vendor lock you and might cause you to spend unreasonable amounts of money, or will let you remain ignorant about how to actually manage the software that you're using, i think SaaSS (Service as a Software Substitute) is a relevant term here: https://www.gnu.org/philosophy/who-does-that-server-really-s...

That said, what works for me and even the company that employs me, won't work for others. And what works for others, won't work for me. This is all because of how different the circumstances of various people out there are: i cannot afford AWS, i cannot afford Azure, GCP and managed services for my own needs.

I currently pay 320 EUR for 6 cloud VPSes per year (and additional amounts for the occasional replacement HDD for my homelab), whereas others pay similar amounts for their cloud platforms of choice per month. For them, depending on their circumstances, it might be more cost effective to spend their time working and throw money at problems, whereas for me it's almost always more cost effective to learn the tech myself.

Similarly to how in Latvia you could hire a team of developers for what one developer would cost in the US. Companies have other factors to consider, of course, but this is just one example - the alternative (opportunity) costs of individuals.

Edit: Of course, some in the comments are talking about hundreds of VPSes/VMs/nodes and in my eyes, that's just an order of magnitude or two higher than what i'm talking about. I've seen plenty of companies in my country running their own data centers and there have been relatively few issues with those that i'm aware of. Something like Ansible and container clusters can scale pretty far!

The problems were more often caused by either mismanaged environments/deployments by developers/agencies who just didn't care about shipping sustainable software but cared more about getting paid and making their software someone else's problem, or making mistakes early in the development and not considering load testing and scalability of the systems as priorities. I'd argue that you can do bad engineering anywhere, though, be it on prem or in the cloud.

* understanding what you're actually doing is better than outsourcing your knowledge.

Most of the issues I see from either on-premises or cloud generally come from not actually understanding the business/use cases/environment. Cloud becomes the 'solution' to a problem of people and process, rather than a value proposition that augments existing reality. You can't make good decisions (e.g. Should I outsource my taxes) if you don't understand what doing those thiggs acrually involves (which most people don't bother to even try).

And usually worked into this is a lot of outsourcing of expertise to the point the business relies on third parties to tell them what to do, which is never a cheaper outcome.

Managing all of these careers is a big cost though in time and effort

What you call "sometimes it's not just cost", actually are costs. Only it's indirect costs that are hard to measure, but these costs definitely exist and definitely have to be taken into account somehow. A large part of hidden costs in corporates is the things you are not doing to save costs because the infrastructure is not flexible enough. I think everyone will recognize this, but it's really hard to put a number on it though.

What? Vertica exists since 2005. New products/services come and go. If that precious data truly has so much value, then it is already being processed in your inhouse system.

These new services are nice for startups and for eventually outsourcing... Aaaand of course for upselling to folks who are already in the faith.

Velocity is and advantage. Yes, but if you really see a brutally good deal with some new AWS service there's nothing preventing you from using it, DC or no DC.

So what you're saying is on premises you need competence to do things but in cloud incompetence is no problem!

These are arguments that wreak of developers who don't see the need to worry about things like 'cost,' 'reliability' and 'efficiency' because of the need to be 'innovative' and breaking stuff faster to win the market.

I'm sure it's fine so long as the developers are really competent.

Is this how people really think?

Correct me if I'm wrong, but Redshift's value proposition was not that it was the first DWH system, but that it was the first one that could scale up and down and be charged by the minute.

> Can you provide a self hosted granular access permission to your RAID?

Yes.

> How hard is it to configure and maintain?

Very few things are harder to configure or maintain than they are on a cloud service, because if they were, someone (e.g. you) would get frustrated and make them easier, and then they wouldn't be for anyone else.

> Will your colo deflect a DDOS attack?

Ah yes, S3 can handle serving that many requests and keep everything online. But then don't you get a bill for $72 billion dollars?

> Can you provide a self hosted granular access permission to your RAID?

This is the second-level mistake engineers commonly make:

The right questions isn't "Can you do X". Give engineers enough time and resources and they can usually come up with a solution to do X.

The real question is "How much time and resources need to be invested to accomplish X at a satisfactory level?"

And the third-level mistake is to assume that getting something to work once is the finish line. In practice, getting something to work once is just the beginning. Getting it to a maintainable, well-documented, repeatable state is a lot more work.

Cloud services make all of this effort disappear. Type a few commands and it's good to go. Now you can take all of the engineering hours that would have gone into the DIY version and allocate them to working on the company's product instead of reinventing architecture that you could have simply paid for.

Good engineers are scarce and expensive. Using them to reinvent infrastructure that can be trivially purchased for a nominal amount is a terrible move most of the time. Even when it does make sense, the right move is to build the prototypes on AWS and then consider transitioning to self-hosted later if the numbers work out.

Umm. why wouldn't you run Ceph yourself? It speaks S3. (It has an component called RGW - Rados gateway, completely stateless, scalable, implementats the S3 ACL Policy xml)

And yes, running it has a cost.

But it's also has the advantage that devs can run it locally in docker easily. CI can spin up endless test clusters.

And so on.

Obviously you are right that the right way to compare cloud vs non-cloud is to look at the full picture. And that also means we need the context.

Small/hobby project? Doesn't matter. You can run on your own toaster or on Oracle cloud or on AWS/GCP/Azure. Just do what you want, the costs are negligible.

Operating business with stable well predicted size? Again, do whatever you want. If IT is a big part and costs matter, optimize for cost and run it on a few dedicated boxes. If you are not cost sensitive and you want to be one of the cool kids run it in AWS or whatever. (We have a client that exists for 25+ years, reached its optimal size, does some innovations from time to time, but it is basically a new website or app. The underlying backend is the same, maybe they'll replace it eventually. Probably with a complete SaaS and then they'll only need to host a landing page.)

Large multinational company with more departments than sanity? Again do whatever you want, likely you have bigger problems than the cloud bill or the inability to run one more app in your DC.

"Unicorn" startup? Crunch the numbers, do what makes sense. Everyone knows that "Netflix went full AWS" but maybe not everyone knows that they went full on-prem CDN more with their hundreds/thousand (s) of local caches at ISPs / IXPs.

And so on.

My availability on-prem is vastly higher than AWS, especially over the past month.

But can they make bandwidth 1000x more expensive?

Not sure where you get “vast majority” from, but as an anecdote I worked for two companies that operated the platform teams at 10%-15% the cost before cloud (including headcount).

However, as I’ve eluded to in other threads of this kind; people don’t like to invest in their own tech, an ideal budget would have been around 15%-20% or cloud spend, we could have solved nearly everyone’s pain with that amount of money.

DR does mean paying more, but just so we’re clear: you have to do DR in cloud too, meaning read replicas of database instances and off-cloud backups which auto restore. If your platform costs more money then this is not a good thing and isn’t just baked into the normal cost.

Ex Amazon here.

I've been in companies that owned their datacenters and it was much, much cheaper than using any cloud service.

Poorly managed datacenters exist but that's an organization problem. Remove the datacenter and you'll have poorly managed cloud instances and services costing millions.

> I know companies in which you have to wait _months_ for a small server to be allocated to your team. AWS does it in seconds

If those companies ever migrated to AWS it would probably still take months to get a small EC2 instance allocated to them. Likely the problem is bureaucracy, not competence.

How many organizations move either between clouds or off cloud entirely? I imagine for the former there’s some import tools in order to poach customers.

But between all the advanced proprietary software solutions and exorbitant egress fees, moving off cloud entirely is rather difficult.

Part of the issue I see overlooked is the cost of acquiring and losing physical assets. If you have an onsite data center and a meteor hits it, how bad off are you?

Disclaimer, I barely passed my aws associates cert and have 0 qualifications to weigh in on this subject with any authority. It's just a point I've seen glossed over before. Yeah your data center is cheaper to run, but is it cheaper to replace?

The issue is that the 3x cost doesn’t include the cost to staff the team that runs your in-house platform software on your colo setup, and the mixed incentives you get when a sizable part of your team is not shipping product but instead shipping say, Kafka queues. My previous employer has one of the largest colo deployments of the f500 companies and it was a mess technically and organizationally. At the end a few hundred infra devs were laid off and projects scrapped after ~24 months of no visible progress on platform stability and support.

Interesting. What workloads are you running there? Internal in-house built line of business apps?

Cost is a huge one, but availability and service priority is a big one for me: Amazon and Microsoft are going to provide service and choose to deploy updates and do maintenance on schedules that work for them, not schedules that work for you.

Your in-house IT department knows not to take a risk the week before your big product launch. Amazon and Microsoft simply do not care about your product launch at all, and probably aren't even aware of it.

You're going to pay for infrastructure and IT staff either way, so why not pay for IT staff and infrastructure that prioritizes your business needs and not their own business needs?