That's when it clicked for me.. comparing my hourly salary rate vs. the cost of running these services "in the cloud." Entirely eliminating "system administration" from my duties was absolutely a net win for me and our team.
... and adding "cloud administration".
What is it with people doing completely one-sided analysis even when they experiment the thing by themselves? Is cloud administration less time consuming than system administration? That's not my experience, so I'm quite interested on how it got so.
The same isn't true for less standard kinds of service. The more standardized something is the easiest it is to decide what to hire, troubleshoot, and learn to configure your options. The less standardized it is, the harder all of those things become. VMs are very standard, email servers are less so, but not by a huge margin. Web accessible disk space and on-demand interpreters are completely non-standardized and a hell to do anything with.
Also, some services do need more upkeep than others. Email is one extreme that requires constant care, file storage and web servers demand much less attention.
Infinitely, and if you look at it from a startup lens it only makes sense. One needs to point only at the recent log4j incident. This is obviously a gigantic black swan event, but even just ongoing security patching at the OS level can be a full-time gig. There is absolutely no substitution for being able to ship code to a platform that just runs it and scales it for you.
Andy Jassey had a great slide a few years back at Reinvent, when talking about Lambda -- "in the future, 100% of the code that you write will be business logic". If you really think about that: how many times have you had to write some kind of database sharding logic, or cache invalidation, or maintaining encrypted environment variables, whatever. That idea that you can toss that -- and what that gives to teams, not having to spend massive timesinks and budgets and hiring and all of that on -- effectively -- solved problems, you really start to understand how you can move faster.
The present reality of Lambda is quite different though. Even though the code of the function itself is more or less "business logic" (although this is a really meaningless term when we're talking about known programming languages and computers), the scaffolding around it with Terraform/CloudFormation/Serverless/etc. is substantial, riddled with quirks and is really time-consuming to figure out and update. I don't think I spend less time on this accidental complexity now when we have most of our logic in Lambda, compared to the times when we were just running Flask apps in a VM.
This is not to mention how hard one has to fight to overcome the limitations of the runtime, e.g. adding some "warmers" scripts to reduce cold-start latency (no, provisioned concurrency doesn't help and is ridiculously expensive). And then comes the bill after you accidentally created invocation loop between two functions.
The scale-to-X and scale-to-zero features of Lambda, along with the guaranteed interface to your lambda with predictable input and output requirements, is incredibly empowering for an engineering team. I can absolutely guarantee that we have spent far, far, far less time maintaining our infrastructure than what we would need to be doing if we had a big-buncha-EC2 setup.
Imagine that the environment issues get taken care of, because Amazon has teams and teams and teams of engineers who are working on just that. Cloudflare has the zero-cold-start isolates. All these platforms are heavily invested in making your development and deployment experience as easy as it can be. Concentrate on writing your code, and you'll reap the benefits.
That's just a cronjob. I know some people don't like doing it that way, but that's on them. I've seen this work for years in production with minimal trouble.
Still, the mitigation was like 4 hours to hunt and disable, and 2 more hours when a full patch came around. Not too difficult.
It was only time consuming for those who worked in Cloud providers where all this crap is centralized and understaffed. In real world scenarios there were tons of teams completely unaffected.
How much business logic is there across all businesses?
I'm aware of a joke at Google about how it could reduce 90% of the employees without any impact on the business.
Immensely more than tech logic, that's for sure.
Just implementing and updating something like a single country's tax code produces more lines of code than the entire Linux kernel, that devs like to brag about :-)
Oh boy, if that's what your server admin guys told you before you went full cloud then I'm sorry, I've got some bad news for you.
If you want to offshore all of your Windows servers, all I can say is "go for it!"
This is off course the highly subjective meaning of a greybeard unixadmin.
You can provision some of these things from cloud providers, but your infra is going to go to shit unless you actually understand what they're really providing you and how to use it. If the only thing you can do is upload a docker image to a cloud provider and click the "create server" button, then that's not really infra work at all. It's like Wix for sysadmins.
While cloud may have a lot of advantages, I don't think it's trivial to run or manage. The AWS dashboard is simply overwhelming. Trying to decide between different, but overlapping services is time consuming. And while you can rebuild somewhat easily, you're also almost certainly going to have to do that as you learn about Amazon's little quirks. Your general RDBMS experience doesn't map to DynamoDB very well and you'll be in for a rough time when you learn you can't just add a new index or whatever.
Then you have all these provider-specific APIs. My experience with both Amazon and Google is that their services will return errors that they claim should be impossible, so you get to have fun debugging that in a service that you don't manage. Your application will invariably add a bunch of handling for exceptional cases and accumulate your best guess about how this impossible situation came about.
Then you have the constantly shifting devops orchestration tooling space and "best practices". I've lost track of the number of times I've needed to pull my Terraform state file and manually edit this gigantic JSON file because some plugin updated an internal struct in an incompatible way.
I'm sure there are people that get an environment up and running in one of IaaS platforms using just the web console. I've never seen it managed that way at any company I've been at. Instead, the devs own the IaaC stuff. It's certainly easier to be in multiple regions that way, but I have a hard time believing any time or money is really saved. Sure, no dedicated ops people, but now your expensive devs have to deal with and probably be on call. Moreover, all that archaic time-sucking Unix admin knowledge acquisition everyone is worried about is just replaced by time-sucking knowledge acquisition of a proprietary service and all its quirks.
Maybe we're talking about different levels of "cloud"? I can buy that Heroku is easier than AWS.
If on a test run or onboarding this application will find just one security hole (e.g. public s3 bucket) you will have the customer lifetime because he‘s afraid he will make a mistake again.
The amount of money set aflame is astounding.
You don’t go cloud to save money. You go cloud to get flexible and reduce capital expense. It’s like leasing a building vs buying. More about tax and accounting.