Hacker News Reader

Comment by jagged-chisel | Hacker News Reader

Forget? You have to hire people for that. We are a software organization. We build software. If we rent in the cloud, there is less HR hassle - hiring, raises, bonuses, benefits, firing … none of that headache involved with the cloud.

Technically? Totally doable. But the owners prefer renting in the cloud over the people-related issues of hiring.

View on HN

This is exactly the rhetoric Microsoft used in the 00's with it's "Get the facts" marketing campaign against Linux and open-source: "Never mind the costs, think about the people hours you are saving!".

It wasn't as simple as that then, at it's still not as simple as that now.

Nope and never has been but to (some of) both sides “it depends” means you are on the other side.

It’s become polarised (as everything seems to).

I’ve specced bare metal, I’ve specced AWS, which is used entirely a matter of the problem/costs and relative trade-offs.

That is all it is.

In fairness to Microsoft, this argument should have been correct. It ought to be possible for Microsoft to offer products with better polish and better support than open source alternatives, and that ought to more than compensate for any licensing costs. Whether Microsoft actually managed to do this is debatable, but the principle is sound enough.

It sort of was especially with respect to desktop software. The licensing costs associated with Microsoft Office etc. were probably not really that much compared to the disruption with switching offices of people who just wanted to do their job to open source alternatives.

This is true, but also really funny considering that even today the average windows sysadmin can still barely use powershell and relies on console clicking and batch scripts. A good unix admin can easily admin 10-100x the machines as a windows admin, and this was more true back in the early 00s. So the marketing on getting the facts was absolutely false.

Citation needed on that one. I've only worked with a minority of Windows sysadmins who are as incompetent as you say. And yeah, of course a good unix admin can run circles around a bad windows one, but the converse is just as true. A good Windows admin can run circles around a bad unix one. It has nothing to do with the operating systems and everything to do with technical competence of the individual.

There are a LOT more bad windows admins than bad unix admins though. The floor of being a unix admin is so much higher that it already filters out a lot of people. There are so many MSPs and small businesses with a windows admin that does everything through a console its crazy. You are right its all about the admin, but on average, the average linux admin is far more comfortable scripting than the windows admin.

This is the fallacy that Amazon sold everyone on: that the cloud has no headache or managment needed. This is manifestly untrue. It's also untrue that bare metal takes lots of management time. I have multiple Dell rack servers colocated in several different datacenters, and I don't spend any time at all managing them. They just run.

> This is the fallacy that Amazon sold everyone on

I’ve been working at a place for a long time and we have our own data centers. Recently there has been a push to move to the public cloud and we were told to go through AWS training. It seems like the first thing AWS does in its training is spend a considerable amount of time on selling their model. As an employee who works in infrastructure, hearing Amazon sell so hard they the company doesn’t need me anymore is not exactly inspiring.

After that section they seem to spend a considerable amount of time on how to control costs. These are things no one really thinks about currently, as we manage our own infra. If I want to spin up a VM and write a bunch of data to it, no one really cares. The capacity already exists and is paid for, adding a VM here or there is inconsequential. In AWS I assume we’ll eventually need to have a business justification for every instance we stand up. Some servers I run today have value, but it would be impossible to financially justify in any real terms when running in AWS where everything has a very real cost assigned to it. What I do is too detached from profit generation, and the money we could save is mostly theoretical, until something happens. I don’t know how this will play out, but I’m not excited for it.

I can confirm this.

The AWS mandatory training I did in the past was 100% marketing of their own solutions, and tests are even designed to make you memorize their entire product line.

The first two levels are not designed for engineers: they're designed for "internal salespeople". Even Product Managers were taking the certification, so they would be able to recommend AWS products to their teams.

As a business owner that pays the hardware bill, what you see as the benefit of your current environment - or a downside of moving to the cloud - I see in a completely different light. To some extent I’d be upset with arbitrary amounts of paid-for capacity just lying around with zero accountability for that spend.

You miss the good time spent debugging a firmware issue, which leads to packet drop on the NIC (or data corruption on the nvme)

I do not miss that crap

> I don't spend any time at all managing them

Who does, then? Even with automatic updates, one can assume some level of maintenance is required for long-term deployments.

Don’t get me wrong, I love running stuff bare metal for my side projects, but scaling is difficult without any ops.

No one. I have automatic backups with proxmox backup server. Updates are automatic and deployments are automated.

Every company I’ve consulted for has hired a team dedicated to just setting up and monitoring AWS for the software devs. Hell, you’d probably reduce headcount running on bare metal.

I have spent about 1 day waiting for every 5 days doing stuff at my last 3 jobs all of which were growing companies thinking that they needed the power of the cloud, but they sure as hell were not paying to make it fast or easy to use.

Pay some "devops" folks and then underfund them and give them a mandate of all ops but with less people and also you need to manage the constant churn of aws services and then deal with normal outages and dumb dev things.

Pretty much this. Most companies have the "devops" folks fully dedicated to maintaining the cloud stuff.

In more than 15 years of experiences, in various compagnies, the number of people who can build and run an on-premise infrastructure sanely can be counted on my right hand fingers

These people exist, but we have far more stupid "admins" around here

When you are not in the infrastructure business (I work in retail at the moment), the public cloud is the sane way to go (which is sad, but anyway)

I help people run their systems.

Clients that use cloud consistently end up spending more on devops resources, because their setups tends to be wastly more complex and involve more people.

I've worked on both kinds of companies in almost 25 years and I can confirm this is true.

The biggest ops teams I worked alongside were always dedicated to running AWS setups. The slowest too were dedicated to AWS. Proportionally, I mean, of course.

People here are comparing the worst possible of Bare Metal with "hosting my startup on AWS".

> The biggest ops teams I worked alongside were always dedicated to running AWS setups. The slowest too were dedicated to AWS.

I wish I could come up with some kind of formalization of this issue. I think it has something to do with communication explosions across multiple people.

Increases in complexity exponentially increases mistakes + MS Teams meetings are just a glorified game of telephone.

Don't make perfect the enemy of the good.

This is a toupee situation. Every effective company I've worked at has a slim platform team that might make some nice company specific templates for how to deploy, but individual teams were responsible for creating and owning their infra. The idea of having an AWS ops team is absurd if you're not at a truly massive company (XX,000+)

I have never, ever seen dev-created infra that was well done, much less with repeatable IaC. It’s always résumé-driven nonsense based on whatever someone read on blogs, and they have no clue how any of it works, only that the output what they expect.

Recently I've seen a lot of IaC created by dev, unfortunately. All vibe-coded, of course.

Could definitely be skewed by my time at AWS and working for companies that hire ex-AWS people, but I've never seen infra being the real third-party roadblock. I've always seen a design review -> IaC creation pipeline that's relatively fast.

The "AWS ops team" is often 1-2 people and often part of the dev team formally, often augmented by external consultancies like mine. You start seeing that kind of structure in teams with 10+ people when the dev created infra starts collapsing under its own weight and they realise they need people with actual ops skillset.

I have literally never seen this at 20, 400, or 3000 engineers. But the companies that I've worked at have all been "name brand" or startups on the path to acquisition.

I've literally seen this at many dozens of companies, including both "name brand" and startups on the path to acquisition.

Same difference. Just because the huge amount of AWS work is distributed among other teams doesn't mean it's less work.

AWS in 2025 is way more work than Heroku/Fly/Vercel, but also more way work than renting bare metal from say Hetzner/OVH, and perhaps even more than renting colo.

This is only true if you're not using value added services. In my experience, teams that don't accelerate by adopting cloud won't use something like SQS or Fargate, they'll throw an MQ on a k8s cluster and get enraged when it doesn't work how they expected.

I literally right now have two customers I'm working to untangle from "value added" services because they've become a threat to the financial viability of their services vs. running their own alternatives.

AWS services are great quality, but they are extremely expensive.

Just because AWS abstracted something doesn't mean you don't need people who understand all the quirks of the black box you supposedly don't have to worry about. Guess what those people are expensive. You also have to deal with a ton of crap like hard resource account limits that on any meaningful size project will push complexity up by forcing you to use multiple accounts.

> We build software

Right, doesn't that include figuring out the right and best way of running it, regardless if it runs on client machines or deployed on servers?

At least I take "software engineering" to mean the full end-to-end process, from "Figure out the right thing to build" to "runs great wherever it's meant to run". I'm not a monkey that builds software on my machine and then hands it off to some deployment engineer who doesn't understand what they're deploying. If I'm building server software, part of my job is ensuring it's deployed in the right environment and runs perfectly there too.

Ultimately these owners hire me to cut their 6-figure AWS bill by 50%. It's mostly rearchitecting mistakes. Amongst them is taking AWS blog propaganda at face value. Those savings could be 80% if they chose managed bare metal (no racking and stacking).

> Forget? You have to hire people for that. We are a software organization. We build software.

You don't need to hire dedicated people full time. It could even be outsourced and then a small contract for maintenance.

It's the same argument you could say for "accounting persons", or "HR persons" - "We are a software organisation!" - Personally I don't buy the argument.

Outsourcing and cloud cost are always underestimated.

> It could even be outsourced and then a small contract for maintenance.

Yeah, those people we outsourced to happen to work at AWS.

They don't though. You still need devops when you use AWS, and most organisations end up needing more time spent on devops when they use AWS.

Until you factor in the legions of devops writing terraform, iam, and cicd scripts.

Forgot? Driving something on AWS needs also a lot of people. In my experience even more. The term SRE did not exist before.

You can just set up your own cloud on leased machines, and pocket the huge difference in cost. Devops languages are pretty easy to learn, IME, and the infra stuff takes less maintenance than the AWS proponents seem to think. I guess it depends on your usage profile, but like bandwidth especially is ruinously expensive compared to what you get with leased machines.

I really dislike the fallacy that just because you're buying something it means that you're not building anything. In practice this is never true: there's always some people-in-your-org time cost of buying something just as much as there's some giving-money-to-other-orgs cost to building something. So often organisations wind up buying something and spending way more time in the process than it would cost for them to build it themselves.

With AWS I think this tradeoff is very weak in most cases: the tasks that you are paying AWS for are relatively cheap in time-of-people-in-your-org, and AWS also takes up a significant amount of that time with new tasks as well. Of the organisations I'm personally aware of, the ones who hosted on-prem spent less money on their compute and had smaller teams managing it, with more effective results than those who were cloud-based (to various degrees of egregousness from 'well, I can kinda see how it's worth it because they're growing quickly' to 'holy shit they're setting money on fire and compromising their product because they can't just buy some used tower PCs and plug them in in a closet in the office')

Don't you have cloud architects and similiar figures already?

Reply on news.ycombinator.com