Hacker News Reader

In general when people mention "cost" as an issue for cloud, they're narrowly comparing what a CPU core or GB of RAM or disk costs on paper between cloud and datacenter/onprem.

This completely misses the business case for cloud, and is why these sorts of analyses don't convince that many business folk, and why cloud use continues to grow fast.

The business case is that someone else is doing the infrastructure management for you, at commodity prices, but you still have complete, direct control over "your" resources and ability to easily automate everything.

This tends to save much more money than any % of your CPU, memory, and disk budget could possibly achieve.

It's sort of like saying, instead of taking cars from your corporate fleet to a repair shop when they break, repair them yourself! Just hire some mechanics, build or rent a garage and equipment, and voila! And also, why are you buying electricity from the electric company anyway? Just build a generator next to your office building! Maybe make it nuclear just like some of the big electric companies!

For many companies, the "just host it in a datacenter" argument is no less silly than these scenarios.

The real business case for the cloud is it replaces capex with opex.

Which is great if you have limited capital (bootstrapping, new startup, etc), or are the CEO of a large company that wants to show better cash flow to earn a larger bonus.

In reality, the idea that the cloud somehow means you can outsource your infrastructure team rarely works out in practice.

At best you’ll get much more expensive developers having to spend time on infrastructure which might look like a couple of fewer salaries to pay, but you’re likely now paying for it through dev time anyways.

The REAL business case is cargo culting.

> The business case is that someone else is doing the infrastructure management for you.

Many business people believe that moving to cloud can reduce the headcount needed for managing the infra, but that is usually not what's happening. You will still need more or less the same amount of people to patch the OS and configure the networking, but with a slightly different skill set -- instead of Cisco IOS commands, they now need to deal with AWS transit gateway.

Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you. There are also third-party companies that offers this service. Redundant power and air conditioning are the responsibility of the colocation provider and those are covered by the SLA. Those data centres do have generators on-site.

That said, configuring you top-of-rack switch is usually not covered by the co-location contract. This need to be done by a network engineer, usually the same person that would manage VPCs.

If you're treating cloud as a sort of 1:1 replacement of hardware infrastructure, then what you're saying makes sense.

But we don't ever "patch the OS." Our cloud provider does that for us. Upgrades and patching are automatic within the maintenance window. We deploy containers, the OS is just a platform layer that the cloud provider manages.

As for needing "more or less the same amount of people", that simply isn't true IME. I've seen startups with whole teams devoted to running systems in a datacenter, whereas cloud-based startups will often have one or two guys, possibly even part-time, dealing with their cloud requirements.

At larger companies, it can be harder to make this comparison, since you'll typically have a mixture of cloud and onprem. But if you look at teams with products deployed in the cloud, they'll often have much more frequent deployment, with better automation, and much less dependence on the always overcommitted infrastructure team(s).

> Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you.

That's not very compatible with infrastructure as code. You're essentially saying "keep doing things the old inefficient way, it's not really that bad."

There are all sorts of failure scenarios that a managed cloud system can cover from completely automatically within minutes, that would require frantic calls to remote hands in the old world. And even when recovery isn't automatic, it may still require nothing more than some pointing and clicking to recover.

> instead of Cisco IOS commands, they now need to deal with AWS transit gateway.

> This need to be done by a network engineer, usually the same person that would manage VPCs.

These aren't comparable at all. Unless a company has some sort of very special networking requirements, they can set up VPCs with some pointing and clicking in the cloud provider's web UI. This is a far cry from the level of knowledge required to manage networks at the hardware level.

I see, so you are using PaaS and I was talking about IaaS offerings. The term cloud is so vaguely defined that it can mean so many things.

If the PaaS works for your scale and budget then it’s perfectly fine. I just want to mention that it is probably no longer the case for scaled-ups.

You're right, I should have clarified that IaaS is exactly the stuff you want a cloud provider to take care of for you, and that's where many of the benefits come from.

But IaaS has been around for nearly 20 years now, and containers for at least 10 years, so I tend to assume people have got that message by now, especially on HN.

Re scale, you can run pretty high scale on managed Kubernetes clusters. E.g. GKE supports up to 15,000 nodes per cluster, hosting over 3.8 million concurrently running pods. Multi-cluster ingress is also supported if you need to run multiple clusters.

And now you need to patch the containers, too!

I had to do that with on-prem anyway.

In addition to the servers.

In addition to patching the routers.

I've never heard of patching containers.

Our build pipelines rebuild container images for every deployment. All you need to upgrade some dependency is to set a version number in the Dockerfile.

There's no "too" because that's the only place we deal with that kind of version dependency. (Not counting dependencies internal to a service, like libraries.)

> But we don't ever "patch the OS." Our cloud provider does that for us.

This is absolutely not true. Unless your cloud services are limited to function-as-a-service and serverless/containerized applications,you are indeed responsible for "patching the OS".

This kind of thing is explicitly covered even by AWS' intro to AWS courses, namely in its shared responsibility model.

> As for needing "more or less the same amount of people", that simply isn't true IME. I've seen startups with whole teams devoted to running systems in a datacenter, whereas cloud-based startups will often have one or two guys, possibly even part-time, dealing with their cloud requirements.

Sorry, your claim is outright unbelievable. I've never seen a company who owned any web application that only had "one or two guys, possibly even part-time, dealing with their cloud requirements." Unless the whole company is only "one or two guys, possibly even part-time", your claim simply is extremely far-fetched and blatantly unbelievable.

> Unless your cloud services are limited to function-as-a-service and serverless/containerized applications,

Yes, I'm talking about containerized applications, deploying to fully managed environments like EKS, Fargate, GKE, Cloud Run. If you're not using containers then yes, it will be more difficult to achieve what I'm describing.

I've been the one part-time guy for three different SaaS startups with funding in the $20-30m range, and globally distributed dev teams. I'm a software architect and dev primarily, so setting up the cloud platform is just a side activity.

For one of those companies, I was literally just a part-time contractor. Once it's set up properly, it should be easy for regular admins to operate, and devs can just follow the templates for configuring a service. It's not rocket science.

All the stuff you're imaging is so complex is exactly the stuff that the cloud lets you delegate to the provider. But you have to be willing to do it, you can't stick to the old way you've always done things and expect the cloud to make anything easier for you.

> For one of those companies, I was literally just a part-time contractor. Once it's set up properly, it should be easy for regular admins to operate, and devs can just follow the templates for configuring a service. It's not rocket science.

It's also a gross misrepresentation of the initial statement. Having a part time employee setting up a few containers to run in A container orchestration system is absolutely not what "setting up the company's cloud requirements" means. It's a blatant attempt to oversell a couple of clicks worth of work as covering a company's whole cloud requirements.

> It's sort of like saying, instead of taking cars from your corporate fleet to a repair shop when they break, repair them yourself! Just hire some mechanics, build or rent a garage and equipment, and voila! And also, why are you buying electricity from the electric company anyway? Just build a generator next to your office building! Maybe make it nuclear just like some of the big electric companies!

I think this is an unfair analogy.

It would be more akin to saying instead of hiring Ubers and taxi's and spending several tens of thousands a quarter, buy a couple of cars for the company. But instead of needing a single person to onboard people onto the company's taxi account and paying the invoice each month, you instead need someone to buy the cars, ensure they're taxed and certified, create an internal hiring system, etc.

For some companies it does make financial sense to have your own fleet of cars, in others it makes more sense to get Ubers.

It's exactly the same as cloud vs not-cloud. If you're not taking advantage of the cloud (e.g. serverless technologies) why pay that premium when a small number of dedicated machines and a simpler architecture/stack will do the job better and cheaper? They're only as difficult to maintain as complex cloud environments.

Do the math and think again.

Infra engineers needed to run your "managed" cloud stack cost much more than generic sysadmins for commodity open-source on-prem solutions.

One problem is those cheap, generic sysadmins usually can't deliver a fully-automated end-to-end SDLC with IaC and automated deployment out of source control. And asking devs to do that without depending on managed services is slow and expensive.

Genuinely feels like there’s a constant trickle of arguments from people looking to do all they can to shift the tides such that their skill set is more in demand. Or maybe it’s just an ideological obsession. I don’t know. I feel the same way whenever the “use C for everything” people catch a whiff of performance mattering and very strongly suggest that the only way to dig ourselves out of this mess is to move back to the One True Language.

In my experience The Cloud™ requires a much larger skillset, ensuring that your ops department will never stop growing.

It also depend on what requirements your business has and who you have available. The skillets are different and they overlap in places, but one way or the other if your business is growing your operations requirements will grow.

I am hugely in favour of AWS for web startups I am involved with because 1) I know AWS very well, 2) they provide hundreds of thousands of euros/dollars of credit to startups (I think all the big cloud vendors do at this stage), 3) it's useful to have architectural flexibility, and 4) they can take on a bunch of the security and availability burden that can otherwise be very labour intensive in a small company.

Also you can pry S3 and RDS Aurora out of my cold dead hands!

I have an impression that is "avoided" in many companies by demanding from the newly hired programmers to have "cloud experience". My problem is that that usually ends up being "programmers have taken courses about the way cloud vendor Y does the job" which then actually means "the programmers had drunk the propaganda of the cloud vendor Y" which is not a directly reusable knowledge for anything than "here's how we adjust our work to be charged by the cloud vendor Y". Which I think is guaranteed to have a company which can't even imagine doing it any other way.

I believe both arguments held some merits. It all depend on the scale.

If you are a startup and/or have a limited/specialized business scope* Cloud can be the best choice.

For organizations of 4000+ people that manage 100+ applications (from different vendors) with a 90+ people IT department this is maybe not the silver bullet. At this scale you need to keep complexity and security manageable and a more traditional redondant datacenter architecture might be less of a headache than the cloud.

*PS: What I mean by "limited/specialized business scope" is for instance Twitter/Meta. Huge scale but heavily specialized around their main products and develop most of theirs application in house.

Yeah, at that scale your devs end up bogged down in cloud concerns. We can't keep anyone at big companies very long because of all the arduous infrastructure bullshit that they have to deal with. Lots of it ends up being done outside of Jira. Total mess.

The other thing they keep missing.

Where are the people with the skills to do this work? I wrote a whole post about it, but it is not like people with that knowledge exist in troves just waiting to be hired.

There are no path to learn this stuff either. Do we really expect every single company out there to build teams of sysadmin and operators from a non existing pool of talents?

It is easy to say that people can move to onprem, but orgs can barely hire the talents they need already. Sure let put even more pressure on a limited supply.

AWS and Azure is a much more niche skill set.

E.g., there's probably three orders of magnitude more engineers familiar with Postgres out there in the world than with DynamoDB.

To the point of running it smoothly at scale in prod with backup?

Nope. Hence you learn the RDS api and use it.

not to mention that the obvious/incremental move is not to completely exclusive DC, but to simply cheaper providers with managed VMs, then dedicated (but still managed) HW, and there's a long tail of that before breaking ground for a new DC makes any sense. somewhere along there's Oxide's "cloud in a rack" thing.

I agree on the security part, and what about the cost argument? I think I've seen a few companies switching over to data centers due to the costs they can save money on moving from the clouds, and I also think mature orgs prefer cap ex and depreciation to op ex (although I am not far from clueless on this point)?

It handwaves all the things EC2 and all services built on top do.

The amount of engineering needed just to migrate droplets live is enormous - it’s an art (according to a friend in hypervisors), and that ignores everything else regarding hypervisors and security.

Scalability is also a thing you get “for free” with serverless computing.

> The conclusion completely handwaves the massive overheads that come with not only owning your own infrastructure

Even if you feel that is a serious obstacle (which I personally don't) you can buy a managed solution (such as an Oxide rack) and use that on prem.

Thanks all for your engagement and comments. I have updated the article to address many of these points.