My gut feeling says part of the problem is C-levels, like CTOs, in recent time, coming solely from programming experience and its just mentally hard for them to even consider having physical hardware (yet on colocation!) and managing such systems. It's just out of equation.
On the article itself, billions scale payments are astonishing. And how much paying for traffic would add here too.
I'd say that if you really want to have hardware on your own, you get the right people to do the job for you. CTOs are not managing that on the floor. So I'm not sure it's the real reason.
I believe in the article they clearly say they just use colo, not running own DCs
> I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already.
This looks strange to me, to focus on software but not on company goals, one of them quite often is to increase profits/decrease spending. Focus on software may happen on lower level of responsibility from my PoV.
No offense, but for me it sounds like someone having talent for bakery and cooking, decided to open franchise of McDonald's, becoming General Manager of that shop and then suddenly says - I wanna focus on burgers only, not hire/fire workers.
Granted, I'm never using edge/runner stuff and choosing the stack from the get go as something performant. I'm also talking about headcount < 100, which I also prefer to stay that way.
In short: it's not the background (I'm not the one directly managing hardware anyway!), but the upside is not enough for the risk, costs and headaches in general. You need to have a real good use case to do CoLo/DC. If you build web software, you probably don't need it.
Not sure I got this point right - if you need to create you own product you will need to hire programmers as well and get right people. Same for PMs, Security, Infra guys as I see it.
SWEs working on a product are part of a different budget (R&D) from IT (Finance & IT).
As a tech company, you can justify high R&D (they build the product) and Sales&Marketing (they sell the product) spend, but you cannot justify Finance & IT spend.
This depends company to company ofc, but for plenty of firms, it's fine to eat the Public Cloud cost because DevOps can be treated as part of the R&D bucket, and there are common metrics that all companies need to follow (some due to experience, others due to inertia).
Then "infra" would be under "IT" in such kind of setups?
From my experience it was bit different - say Infra, Dev, Security departments having their own Head Of Dev/Infra/Sec reporting to the CTO and having their budgeting aligned with CTO. CorporateIT being a bit different thing. Procurement may be shared among all of them of course.
Not saying that's the only possible setup of org structure - just what I've used to more.
Depends what you mean by "infra".
The team that would manage your DC (either self managed or CoLo) would generally be under the CFO or COO (who will generally report to CFO), and as such that team would fall into the Finance procurement bucket.
Even if R&D has budget for multiple racks in your DC, it will still cost money on the Finance&IT side of the house to staff people to maintain it and manage it.
This model is very clunky, because procurement takes time, and this lack of nimbleness can drastically affect delivery roadmaps.
CFOs also don't like this model as much, because their job is to manage cash flow, keep books and accounts ready for audits, build financial projections, etc and time consuming work like managing IT procurement (while critical) is a major slog.
The model you mentioned is more common now (Platform/Infra team acting as an interface w/ Eng), but even they would offload DC and CoLo management to the CorpIT team.
Core IT (not Infra - Infra often means DevOps/Platform which is part of R&D), like any other shared function, is placed under the CFO/COO org because it is a shared internal function that's used to keep the lights on.
These are functions that by definition are a cost center, and as such R&D really doesn't want to own it, as it distracts from the core mission of delivering forward facing features.
The whole point of the DevOps/Platform Eng model was to incorporate operations under R&D, but having R&D own and manage physical assets is a major distraction, and a major reason Public Cloud became popular among plenty of companies.
Cost of compute isn't the only variable that comes to play - depreciation, hiring/firing, security, redundancy planning (because these are physical assets), etc are all significant headaches operationally and financially.
Technology is a tool - you choose the tool that works best. In some cases Public Cloud is the solution, in other cases it's a CoLo. It comes down to your specific organization's needs.
When you have a tiny Corp IT, almost invisible in a company's annual budget, and nobody in the company wants to care about it (a lot of pain but zero rewards) -> it goes to CFO. You have Corp IT as a major cost center which is bigger than all the other spending altogether (mature IT companies) then you want someone knowledgeable and trustable to run it, usually CIO.
Sometimes, Corp IT as internal infrastructure supporting employees, development, backoffice is under the same person running production for the clients. Then standards of the internal IT and external services are the same (and very hard for internal services to achieve).
In other cases, production and corp IT are different departments with not so funny unresolvable collisions, like buying incompatible hardware for testing environment (internal services) and production.
I can see why, though. First, there was so much money flowing into IT since 20-ish years that cost was not a consideration both on infra and salary side. Second, connecting all the piping gives an illusion of doing work and feeling "important". You know, we need to be scalable because Google does, and they're successful, right?
I partially agree with you, but my point is - things are even worse at times - arguing happens when there is more than single person involved - CTO may just not even considering/thinking on using hardware, it's a very strong habit to relay on cloudy stuff, atrophy of some sort.
> Second, connecting all the piping gives an illusion of doing work and feeling "important".
This happens as well, but I think it's more about "keeping things under control" - humans feel more comfortable when they control or tend to believe they control things, all humans, not just developers. From this perspective "gee, this Cloud has API and Console, I can manage it" gives this assurance -> leads to have feeling of control => preferable solution.
Around a dozen years ago, my business was designing building and supporting physical infrastructure for startups.
One company was humming along nicely on $4000 of used hardware and a $2000 a month cage in a Colo facility.
Their business had ramped up and got some funding, so we got them another 60k worth of hardware in nother facility.
They onboarded still more customers to where they needed a few thousand dollars worth of SSDs to keep up with their random io demands.
But their new VC-installed CTO was like… No! We spent all this money on hardware and it’s not doing what we need it to! That’s crap! We’re gonna move to the cloud and save money.
So they moved to Amazon.
Their first month’s bill was $50,000. And it only went up from there.
The cloud is dandy for small workloads. But where you’ve got a consistently large workload the break even point on owning your hardware is a lot lower than most people think, even factoring in infra management expenses.
Largely the point I'm trying to put here
> Their first month’s bill was $50,000. And it only went up from there.
Well, I've read it's not uncommon to have salaries of 200-500k USD/year per programmer, on that scale may be it doesn't matter is it 50k spending on infra or 100k . For others such extra spending is _something_ though.
Individual contributors see these numbers and act like it's some huge amount of money but in the grander scheme of things when you're approving budgets for multiple teams of senior SWEs and other roles, it doesn't even register.
So, ICs who sweat that kind of waste are right to do so. If management sees that someone thinks 50k/mo is no big deal to squander, bet your ass there’s a good chance they’ll take at least a month’s worth out of that person’s annual bonus since that kind of money matters so little to them.
Source: I was there
When you’re a growth stage company, the flexibility and leanness that comes from the cloud is important. Once your core product and computing demands stabilize, it’s a good time to figure out what makes sense to move to cheaper compute.
Anyone running >1000 servers with >80% utilization should consider going partially colocated.
Comma.ai and Tesla come to mind who run their own training clusters.
If you are serious about compute and cost of compute and storage is ~100s of engineers, then moving off cloud is a wise strategy.
Cloud providers have huge margins. One can run a very profitable business if they get a pie of that margin.