All it takes is bad support, severe price undercutting, or degrading service and we're off to another, or a split. By having people at least feign caring, giving support, and giving suggestions, they're locking us in and getting way more back in revenue than what it's costing them, at least from us.
The argument for 6 figure cloud spend was always "it'll help you not need as many infrastructure-related dev ops engineers", right? Would you say that's true for your org?
Every org I've ever worked at has had 6 figure monthly cloud bill, and the 6 figure monthly payroll bill related to teams working with cloud infrastructure
Ok, let's make one thing patently clear: ITS THE SAME IN THE CLOUD
All the cloud vendors will tell you need to have stuff replicated in multiple "Availability Zones" or "Regions".
And yup, the nickle & diming nature of the cloud means that's going to cost you double or triple.
Bandwidth is the one thing where the cloud clearly wins wrt idle servers... except that DC bandwidth is a hundred times cheaper than AWS bandwidth, so you should prefer buying 133% or 150% or 200% DC bandwidth by a mile.
You can redirect traffic to another region and have autoscaling spin up EC2 instances, etc.
But, very notably, you can have a suspended cloud backup even if your main servers aren't cloud. And the added complexity for datacenter-to-cloud HA doesn't have to be significantly higher than the cloud-to-cloud version.
A stable company where growth as a percentage isn't likely to be significant can run things cheaper on their own in most cases. At least if you consider the cost of the inevitable departure from the cloud provider either to switch another or to go on-prem. And if you aren't willing to make that exit, you can guarantee your cloud provider won't stop cranking up the fees until the threat of you leaving surfaces.
You basically run the base load in your own data center and the surges go to the cloud. My university is evaluating this because sometimes you have multiple labs that need a lot of compute resources at the same time and local compute cluster has finite capacity.
But hopefully you don't get trapped in the cloud and can claw the margin back.
If you complicated your architecture enough - and the cloud makes it oh so easy to make rube goldberg architecture - keeping many different services running or even developing in-house can take a lot.
And it's not like cloud costs you zero in ops work either, just need different set of skills.
But it is not like on-prem stagnated - there is plenty automation in that space too. Our team of 3 manages 7 racks of servers and few dozen projects on them (anything from "very legacy" to 30+ node k8s cluster and ceph storage with it) and the hardware management still isn't majority of our work
"Can" sure... likely to for most orgs no. Certainly not for someone running a 5 figure cloud bill...
as a reference, our standard compute server we use on prem is $12,000 including a 5 year support contract.
> At a certain point, you might outgrow someones else's cloud and build your own,
If your buying $100K servers, and $100K network devices, I pretty sure you are at the point
i've been on both sides of the fence and it's a case of 'grass is always greener on the other side'. the truth is, running any sort of non-trivial infra is ducking expensive.
Precisely zero have been put out of work.
Actually, seems like managing k8s is an order of time expenditure greater than managing an old-school F5 with a bunch of Unixy web servers behind it.
Managing a switch and servers is a piece of cake compared to managing k8s, IMO.
I come from this old world of managing switches and servers and today we definitely need a lot less people to run code in production. I used to work at a company with ~2000 machines in physical data centers before containerization, this required a huge infra team - I'm sure that today I could support the same workloads with half the team.
Devops gives the benefit of cows instead of pets, and a ton of reusable work. So I get why it's more 'valuable' from a remuneration basis.
or Nagios...
(I'm not saying dotnet allowed them to get by on N low digit servers. I'm saying those folks are atypical)
It’s interesting trivia to be sure, but I’m wondering if you were making a point with that
> I'm saying those folks are atypical)
Calling "write not shit code in language that's not dog slow" "atypical" is sad state of our industry.
Also in many cases you can get a lot out of caching if you do it smartly.
Perhaps we are designing far more complicated solutions now to leverage these cloud services, whereas having the constraints of a self operated data center and infrastructure necessitates more ingenuity to achieve similar results.
We used to do so much more with just a few pieces of infrastructure, like our RDBMS's, as one example. It was amazing to me how many scenarios we solved with just a couple of solid vertically scaled database servers with active-active failovers, Redis, an on-prem load balancer, and some webservers (later, self hosted containerization software). We used to design for as few infrastructure pieces as possible, now it seems like that is rarely a constraint people have in their minds anymore.
As an example I have seen many times people attempt to find a reason to use k8s because the industry says they should instead of looking at what they need to do and then determining if k8s is the best for that application
Nothing about "ingenuity", just plainly having some friction in implementation makes for simpler designs.
If you have zero cost (aside from per request pricing but that's not your problem right now, that's management) to add a message queue to talk between components, now that's a great reason to try that message queue or event sourcing architecture you've read about.
And it works so "elegantly", just throw more stuff on queue instead of having more localized communications. We don't worry about scaling, cloud worries about it(now bill for that queue starts to ramp up but that's just fraction of dev salary, we saved like a 2 weeks of coding thanks for that! Except that fraction adds up every month...).
Repeat for next 10 cloud APIs and you're paying at every move, even for stuff like "having a machine behind NAT". And if something doesn't work can't debug any of it.
Meanwhile if adding a bunch of queue servers would take few days for ops to sort monitoring and backups on it, eh, we don't really need it, some pubsub on redis or postgresql we already have can handle stuff that needs it, and rest can just stay in DB. This and that can just talk directly as they don't really need to share anything else on queue, we just used queue to not fuck with security rules every time service needed to talk to additional service.
IF you're pushing 400+ Gbit, sure. Most won't.
For something like our HFT firm, we see high sustained utilization so we have an on-prem cluster of nodes that we run. They're faster (more powerful CPUs), have more RAM, and you can run fiber optic interconnect that's, in practice, faster than on AWS. We still use AWS extensively, though.
I get that if you have management that won't keep even 10% spare capacity online you can't work like this, but they will also "tightly control cloud spending", won't they? Not sure.
Cloud only really makes sense at really small scale.
So, for example, things like PXE boot, which would be useful in your own datacenter, but don't work (well) in clouds s.a. AWS become mystery. Obviously, this isn't limited to just PXE.
And since public cloud's goal is to provide service to as many customers as possible while having as few staff infra engineers, the total expertise among general public disappears. Having worked in the infra business for a while -- it's increasingly more difficult to hire infra engineers.
I know that when I was hired, my job was advertised for over a year, if not longer. They pay may not be stellar, but it's still pretty decent...
In my previous position, I was also responsible for screening and hiring, so I also had some insight into what's available on the job market. My impression is that the younger generation simply doesn't even know this kind of job exists. Those who go into programming aspire to be a Web developer, or an ML developer, or a game developer... DevOps is one of those things people have heard about. But, this label was coopted by cloud some time ago. So, those aiming for DevOps role think they need to learn how to use public cloud API...
It's actually quite eye-opening to have access to the hiring process. One of the very surprising things to me was that newer generation of programmers tends to be confused about persistent / volatile storage. So, if you asked them to give a description of what a "file" is, more than once I've heard an answer like "a portion of memory". Which, while true in some sense, is a weird way to put it. But then I realized that people who are entering the field today may not have been exposed to PCs. Naively, if anyone asked me to show where the files in the computer live, I'd get them a picture of a hard drive. But people who only ever used laptops probably don't even know what a hard drive looks like.
Finally, it's also a chicken and egg problem. You cannot really learn about infra outside of a big org that has its own physical infra. Similar to how you'd never be exposed to configuring BGP if all your exposure to computers is your own PC. A popular method of learning today is to learn online. But, if you do that -- you are almost inevitably driven to the public cloud, because that's the way to provide you with all that virtual infrastructure online. You cannot also practice at home, since the hardware to practice on tends to be very expensive and not very useful for individual users.
Other fields get it easier because they have to select from somewhat prepared candidates. When it comes to infra, it's a lot more investment into teaching one.
It was ever thus. Previously it was CISCO Solution Specialists and HP/UX Most Valuable Engineers and the like, and all the knowledge beyond the basics was completely tied to the vendor. In enterprise IT, it's still like that for Microsoft.
Or is it that there are orders of magnitude more internet-based companies today, all competing for the same talent (which, indeed, is a limited skillset because these are not generally applicable skills). How many of these new companies exist _only_ because cloud infra is available now?
Another factor is promotion into management. Programmers have very little to look for in terms of professional growth. Those who "make a career" tend to go into management. It's so to the point that if you were in the industry for 20 years and you aren't a head of a department at least, you are seen as a loser. So, a lot of those who did know their trade decade or two ago are ticking boxes in Excel today.
Yet another aspect is the nature of venture capital investment into startups which encourages very wasteful short-term solutions. So, while it may be a more financially sound strategy to do things on-prem, the investments are rationed in such a way that there's no money to build the infrastructure upfront, even if in the long run this would've been a better financial strategy.
Similar, I forgot how to config HA proxy and setting up Nginx would take me a few days to re-learn/optimize but I don't need those skills anymore. Just like I don't need to know how to install oracle/pgsql from scratch, setup raid on my computer, compile the kernel, etc to do my day job.
I'm also not debugging the assembly coming out of the compiler anymore looking for compiler bugs (the one time I've done that in my 25 year career, I just pinged one of the eng who worked heavily in that area and he filed a bug to sony 10 minutes later and told me to just add some int ++ / -- and that shifted the compiler bug out of the way. Sony fixed it a month later.
Nor am I a wizzard at add/remove drivers from win98 or setting up a token ring lan.
Germane knowledge comes, and goes. My job isn't to know PXE boot or any other specific tech, it's to ship product...
First, cloud vendors don't really invent the infrastructure from zero. They typically adopt existing solutions to their needs. And they aren't infallible. Every now and then there'll be a bug or an edge case they didn't handle. Alternatively, there will be weird restrictions that you, the customer, would have to go fish for in substantial volume of quite opaque documentation in order to protect yourself from encountering those unfortunate edge cases.
I don't know if AWS internally uses PXE boot. But, suppose they did: this would mean that there'd have to be some specific details of the network configuration that enabled the use of PXE. Also, there could be some pathological cases of interaction between "special" (boot, bootloader) partitions and PXE boot. You wouldn't normally care in VM-based deployment about boot partition as some VM players can even offer to boot your kernel w/o you having a bootloader in the image, so, you may go for a long time w/o ever discovering what your bootloader is configured to do and how it may interact with PXE boot.
Suppose at some point AWS upgrades something in their PXE server / code it runs on clients... and it breaks your stuff. Because this would affect most likely all of your VMs, the effect may be devastating...
But... you didn't even know PXE boot existed. You had no idea that you had to look for some subsection X.Y.Z in the appendix to the user manual that warned you against using a particular configuration in your bootloader, which stayed dormant for a long time.
It's not hard to imagine how this might go financially...
Second, any solution purports to sell to many customers -- the more the better, that's how the solution becomes profitable. Naturally, the solution provider wants to find as homogeneous group of customers as possible, so that with as little effort possible they'd be able to cover most customers. But the reality tends to work against this desire by creating customers who aren't like each other. So, some generalization must happen, and it means that very specific, very individual desires of customers will likely not be covered by the service provider. Now, suppose you, the client, know how the underlying technology works, and thus are able to defeat the unwillingness of the provider to accommodate your very specific needs... or, you don't. Typically, this ends up being not a deal-breaker, but an inconvenience. Often times it's the money you'll pay for the service you don't need, or the resource consumption that could've been avoided.
I cannot think about a good example with PXE, but here's something I crossed paths with recently at my kid's birthday party: I met a guy who works at Gitlab, and that reminded me about a grudge I had against Gitlab for some time. So, Gitlab is an example of service that replaces in-house IT / Ops who'd manage company's repository in this case.
Gitlab, at least initially, gained popularity due to CI bundled into service. This CI knows how to run jobs. These jobs produce logs. The logs can be obtained through API, if necessary. But... if you do this at the time the job hasn't finished, there's no API call to enable paging of logs. So, if you want to display logs as they are being generated, you poll, and each subsequent response will get you 0..N characters, with N ever increasing. This is hugely wasteful, and, in part the reason why their logging configuration puts a hard limit on log size... Well, bottom line, their logging API is bad. But, you won't abandon the service because the logging API is bad. You'll suck it up, especially since they give you a bunch of free stuff... You'll just pay a bit extra, if you don't need the free stuff.
When it comes to software amazon again has millions of machines, regions, availability zones. AWS is very very slow and deliberate about how they roll out software. They do things by Sub AZ (shard, cell), AZ, Region... they can and do split on other axes as well. Rolling out a PXE boot change would be caught by increase in failed boot or under utilized instances (they didn't boot right) pretty quickly in a roll out and rolled back.
All of this is aided by many may of AWS's customers treating servers like cows not pets, so whacking and restarting a small % of hosts isn't super detrimental and likely won't even be noticed. AWS can be extra cruel to internal teams actually. We have have a few concepts internally that push users to be even more ephemeral than normal.
As for how do people learn PXE boot? Experience or on the job training. I have no idea if AWS uses PXE boot anymore, I suspect they have a specialized setup, boot vms directly, etc. There's hardware security with the hypervisors you don't get on normal machines. I PXE booted a linux desktop the other day in the office to install linux so the corp side does use it.
I'm not really sure what the point you are getting at is tho. The cloud providers are doing things on a scale most people are not used to thinking of and the problems are often quite different as are the solutions. Are you rolling your own custom hardware with FoxCon (https://www.importgenius.com/suppliers/foxconn-aws)? Do you roll your own custom silicon (https://www.amazon.jobs/en/landing_pages/annapurna%20labs)? Would you be solving PXE boot issues if you had these capabilities?
There was a point not too long ago when I managed a 6 engineer team responsible for 7 figures of monthly spend. Still seems outlandish to me despite the ops burden being pretty low once we had things sorted.
> By [avoiding those bad things] they're locking us in
Providing a good product at a good price is "locking in" ?
Leaving really involves running two systems in parallel for a bit and gradually doing a changeover - blue-green in production between two different clouds. Which is not cheap/free either, you are actually going to increase your costs significantly as you leave.
[1] tagline: “Migrate or transport exabyte-scale datasets into and out of AWS”.
[2] “Q: Can I export data from AWS with Snowmobile?” A: No. in https://aws.amazon.com/snowmobile/faqs/#Using_Snowmobile
- the Big Three have data centers all over the world. No other provider can match that. (Obviously: if 99% of your customers come from the EU, Hetzner, OVH and friends will be cheaper)
- there is a ton of prior experience in managing and provisioning infrastructure on the Big Three. You got Terraform providers covering every oh so tiny resource they have, and for literally every problem you got dozens upon dozens of stackoverflow/serverfault posts.
- the Big Three have allll sorts of managed services. No matter if you need cheap blob hosting, virtual servers, bare metal servers without paying setup fees, AI training servers, VPNs, global load balancing/caching, data storage/archival, databases, "serverless" hosting, email, SMS, IoT communication, Active Directory and other identity/authentication/authorization solutions, business data analytics, whatever - it's all a one-stop-shop. No dozens of vendors, bills, payment schedules, GDPR processing agreements, budget issues - one vendor, one bureaucracy, everything you need.
It's really bad that the EU never woke up to the rise of the Big Three and say, for example, financed the development of OpenStack to the tune that it could be used by domestic providers, because now the Big Three have all but eliminated the smaller competition.
We're not as cheap as Hetzner or OVH, but we're just as global as public cloud and will match or beat the latency.
And some of those rules already have mandatory entries, so you'll likely only have around 7-8 actually usable.
"Just use the firewall in your OS" is a workaround, but doesn't fully cover the same scenario's. That's why AWS has Security Groups after all.
This limitation means we have to use smaller sized servers (64/128GB ram), rather than bigger ones which would otherwise be more cost effective.
It's a real pita for us, and Hetzner have shown no interest in ever addressing the problem. :(
That being said, it's all still a lot cheaper than AWS, as Hetzner doesn't charge for bandwidth with their dedicated servers. :)
AWS spend = 500k * 12 = 6M
AWS spend / Total spend = 6%
Where the hell does 0.1% come from?