Hacker News Reader

Comment by SiVal | Hacker News Reader

Yes, but what I'm not seeing in their analysis of variable-cost cloud vs cheaper fixed-cost colo is the huge fixed cost of the personnel to manage it. Yes, I agree that switching my food-delivery business from calling for an Uber whenever I get an order to buying my own car can be much cheaper, but if I leave out the cost of hiring a full-time driver for my new car, my analysis is...incomplete.

Your pitch emphasizes features that make servers easier to manage, presumably lowering (but not eliminating) the cost of personnel, so you're obviously aware of this, but I'm not seeing any "net of additional, fully-loaded personnel costs" in their analysis.

It should still be cheaper above a certain scale, but the breakeven where it will make sense to "repatriate infrastructure" is much higher if you include hiring people, and will go even higher if the cloud providers run the numbers and match most of the savings with scale-based price breaks.

View on HN

I don't know how every company does it. But I led the datacenter and networking teams at Square from 2011 to 2017. In that time we went from 1 DC cage with 4 server racks to 4 US DCs, one in a Japan and a couple of network pops. A bit more than 100 server racks total.

My team had 2 guys doing SiteOps. They would travel to the various DCs in the Bay Area and Virginia and do all the maintenance, new installs, etc. And sometimes we'd lean on the colocation remote hands to do a few things.

We had about 5 network engineers, that also handled the corporate network. (12 offices and a network backbone that connected east and west coast DCs, offices, etc).

And maybe 2 SWEs who handled things like our host OS install system, etc. Basically the next layer above the hardware.

So 9 people that were required to run all of that stuff. But really, NetEng ended up spending like 70% of their time on corporate network things because we'd add offices faster than datacenters.

So if we focused on production only, we really needed about 6 or 7 people total.

I did the math a few times (every single year) and compared our costs, including people, to the costs of moving to AWS 3 year reserve instances.

Doing it ourselves was always half the price.

Of course the difference here was we built on-prem from the start. So there was no repatriation that had to happen.

Since then, I've been responsible for large cloud infra on all 3 major providers and learned a lot about what kind of discounts you can get when you're in the double digit millions in annual spend.

I still think, at a scale of single digit thousands of servers, you'd be cheaper on-prem, fully loaded. But admittedly, I haven't run the numbers since 2017.

The problem is not pure cost with a competent team: It's that many companies can't even tell that the team they have is competent. And when the team is not competent, prices, and opportunity costs, are so much higher than what you describe. Imagine a world where adding the equivalent of a 4 core VM with 8 gigs of ram is going to take 6 months from the time a team say they need it, until it's actually available. The meetings, the arguments and other minutiae (like trying to get things installed on the server, because of course you need to play telephone games to get a single command run in a server) make the costs of the server look small.

And those companies are in situations where either people have no idea that their teams are performing badly, or lack the political will to do something about it. Adding the services in AWS is just easier, and might be worth every penny.

When you go to a high performing firm and explain the procedures and decisions of a low performing one, people think you must be lying, but it's true: Broken IT departments are everywhere in enterprise companies, and paying double of what a good team would take to do the right thing is a bargain, given that they can't even get a whiff at a good team.

> Imagine a world where adding the equivalent of a 4 core VM with 8 gigs of ram is going to take 6 months from the time a team say they need it, until it's actually available

If this is the case then the CTO knows that something is very wrong.

I don't think your scenario of companies not knowing whether their teams are incompetent makes sense. You can apply this logic applies to all functions.

GP “kelp” is the Way, but “hibikir” is the Truth.

> at a scale of single digit thousands of servers

On the other hand, at merely single-digit servers, I'm guessing the fixed cost of personnel referenced in the GP comment would dominate.

This is the way.

But most firms don’t know how to employ a you to show them this way.

Reply on news.ycombinator.com