Bare-metal solutions save money, but are costly in terms of development time and lost agility. Basically, they have much more friction.
I guess before Amazon invented "the cloud" there wasn't any software companies...
So it's a fact that for most use cases it will be significantly easier to manage than bare metal.
Because much of it is being managed for you e.g. object store, databases etc.
Setting up Garage for obj store: 1 hour.
Setting up Longhorn for storage: .25 hour.
Setting up db: 30 minutes.
Setting up Cilium with a pool of ips to use as a lb: 45 mins.
All in: ~5 hours and I'm ready to deploy and spending 300 bucks a month, just renting bare metal servers.
AWS, for far less compute and same capabilities: approximately 800-1000 bucks a month, and takes about 3 hours -- we aren't even counting egress costs yet.
So, for two extra hours on your initial setup, you can save a ridiculous amount of money. Maintenance is actually less work than AWS too.
(source: I'm working on a youtube video)
Because there is a world of difference between installing some software and making it robust enough to support a multi-million dollar business. I would be surprised if you can setup and test a proper highly-available database with automated backup in < 30 mins.
Installing software is exactly what multi-million dollar companies do.
Backups are not hard either. There's many open source setups out there and building your own is not that complex.
Oh sweet summer child... let me know how that goes for you.
I guess you never setup one, because I’ve see numerous attempts at this and none took less than a month to do.
It takes about 10-15 mins to apply the configs and install the stuff, then maybe another 10-15 minutes to run a simple test and verify things.
Also, with EKS, you get literally all of this (except Cilium and Longhorn, if you need that, which you don't if you use vpc-cni and eks-csi), in ~8 minutes, and it comes with node autoscaling, tie-ins into IAM and a bunch of other stuff for free. This is perfect for a typical lean engineering team that doesn't really do platform stuff but need to out of necessity and/or a platform team that's just getting ramped up on k8s.
You also don't need to test your automation for k8s upgrades or maintain etcd with EKS, which can be big time-savers.
(FWIW I love Kubernetes and have made courses/workshops of exactly this work)
> how about controlling who has access to servers
It depends on where you want to control access to and how. If you are talking about SSH, you can bind accounts to a GitHub team with about 20 lines of bash deployed to all the servers. Actually, I have a daemonset that keeps it updated.
Thus only people in my org, on a specific team, can access a server. If I kick them out of the team, they lose access to the servers.
I have something similar for k8s, but it is a bit more complicated and took quite a bit of time to get right.
> tamper protected activity logs?
sudo chattr +au /path/to/file
will force Linux to only allow a file to written to and read, but only in append mode and deletions are actually soft-deletes (assuming you are using a supported filesystem). Things like shell history files, logs, etc. make a ton of sense to be set that way. There are probably a couple of edge cases I'm forgetting, but that will get you at least 80% there.
I'm not suggesting that this isn't a viable solution, but I would prefer to be well-prepared and have a team of experts in their respective fields who are willing to have an on-call duty. This team would include a specialist for Longhorn or Ceph, since storage is extremely important, one to set up and maintain a high-availability PostgreSQL database with an operator and automated, thoroughly tested backups, and another for eBPF/Cilium networking complexities, which is also crucial because if your cluster network fails, it results in an immediate major outage.
Certainly, you can claim that you have sufficient experience to manage all these systems independently, but when do you plan to sleep if you're on call 24/7/365? Therefore, you either need a highly competent team of domain experts, which also incurs a significant cost, or you opt for cloud services where all of this management is taken care of for you. Of course, this service is already included in the price, hence it's more expensive than bare-metal.
> you either need a highly competent team of domain experts, which also incurs a significant cost
You can build this team and domain knowledge from the ground up. Internal documentation goes a long, long way too. From working at bare-metal shops off-and-on over the last decade, the documentation capturing why and how are extremely important.
> you opt for cloud services where all of this management is taken care of for you
Is it though? One of the biggest differences between a bare-metal shop and a cloud shop is when shit hits the fan. When the cloud goes down, everyone is sitting around twiddling their thumbs while customer money flies away. There isn't anything anyone can do. Maybe there will be discussions about how we should make the staging/test envs in more regions ... but that's expensive. When the cloud does come back, you'll be spending more hours possibly rebooting things to get everything back into a good state, maybe even shipping code.
When bare-metal goes down, a team of highly competent people, who know it in-and-out are giving you minute-by-minute status updates, predictions on when it will be back, etc. They can even bring certain systems back online in whatever order they want instead of a random order. Thus you can get core services back online pretty quickly, while everything runs degraded.
Like all things in software, there are tradeoffs.
Should you require greater than 99.95% uptime for particularly critical operations, then opting for a multi-region approach, such as using multi-region storage buckets, is advisable. It's also worth mentioning that I have never experienced a full 8 hours of downtime at once in a given year. It has usually been a case of increased error rates or heightened latency due to the inherent redundancy provided by availability zones within each region. Just make sure your network calls have a retry mechanism and you should be fine in almost all cases.
The cloud is much more mature as is dev-ops in general, these days, but major outages still happen. If you run your own cloud, you'll still have major outages. You can't really escape them.
If you have the expertise or can get the expertise, to do it in-house, you should do it. Just look to the US and its inability to build an in-expensive rocket, or even just manufacture goods. They outsourced everything (basically) to the point where they are reliant on the rest of the world for basic necessities.
You gotta think long-term, not short-term.