Let’s write our own container orchestrator though, because control planes are dumb?
With a simple, predictable workload --- what they have --- it can make sense to lean towards static scheduling, rather than dynamic schedulers. K8s and Nomad are both dynamic schedulers.
This is pretty basic stuff; it's super weird how urgently people seem to want to dunk on them for not using K8s. It comes across as people not understanding that there are other kinds of schedulers; that "scheduling" means what Borg did.
And k8s scales very well to very low and high numbers.
Because k8s provides battle tested Features like rollout lb etc.
And the ecosystem is great.
Certmanager, argocd kube stack.
I'm baffled tbh how they had such a difference experience with k8s than I do
Which is the primary reason why we did investigate k8s on-prem. We had already done the work to k8s-ify the apps, let's not throw that away. But running k8s on-prem is different than running your own k8s in the cloud is different than running on managed k8s in the cloud.
Providing all of the bits k8s needs to really work was going to really stretch our team, but we figured with the right support from a vendor, we could make it work. We worked up a spike of harvester + rancher + longhorn and had something that we could use as if it were a cloud. It was pretty slick.
Then we got the pricing on support for all of that, and decided to spend that half million elsewhere.
We own our hardware, we rent cabs and pay for power & network. We've got a pretty simple pxeboot setup to provision hardware with a bare OS that we can use with chef to provide the common bits needed.
It's not 'ultimately flexible in every way', but it's 'flexible enough to meet the needs of our workloads'.
Bare vanilla k8s or k3s is nice but it doesn't do much outside of your homelab. Once you want k8s on production in the cloud you have to start about thinking of: - loadbalancing and ingress controller - storage - network - iam and roles - security groups - centralized logging - registry management - vulnerability scanning - ci/cd - gitops
And all this is no less complex with k8s than with nomad, bare docker or whatever they chose. And definitely no less complex because it is on a major cloud provider.
Ingress, lb, storage, network...
And I have my small setup running with all of it too. Took me a weekend to set it up.
Rke2, nginx ingress, classic lb in front, cert manager and everything else in argocd.
K8s is NOT easy, and I do not believe that in it's current form it is the pinnacle of deployment/orchestration technologies. I am waiting for what is next, because the pain that I have personally experienced around K8s that I know others are feeling as well does not make it a perfect solution for everything, and definitely not usable for others.
At the end of the day it's a tool, and it is sometimes difficult to work with.
It is usually a game of finding the correct spaghetti(log) in a full plate.
And I think it's the best thing for infra since pre cut bread.
In what issues did you run?
The initial deployment for EKS requires multiple plugins to get to something that is "functional" for most production workloads. K8s fails in spectacular ways (even using Argo, worse using Argo TBH) that require manual intervention. Local disk support for certain types of workloads is severely depressing. Helm is terrible (templating Yaml... 'nuff said). Security groups, IAM roles, and other cloud provider functions require deep knowledge of K8s and the cloud provider. Autoscaling using Karpenter is difficult to debug. Karpenter doesn't gracefully handle spot instance cost.
I could go on, but these are the things you will experience in the first couple days of attempting to use k8s. Overall, if you have deep knowledge of K8s, go for it, but It is not the end-all solution to Infra/container orchestration in my mind.
I fought with a workload for over a day with our K8s experts, it took me an hour to deploy it to an EC2 ASG for a temporary release while moving it back to K8s later. K8s IS difficult, and saying it's not has a lot of people questioning the space.
The way I see it is it starts off easy, and quickly ramps up to extremely complex. This should not be the case.
I worked at a company that had their own deployment infra stack and it was 1000x better than K8s. This is going to be the next step in the K8s space I believe and it may use K8s underneath the covers, but the level of abstraction for K8s is all wrong IMO and it is trying to do too much.
The main issues we faced with over 700VMs were: outdated os, full disks, full inodes, broken hardware, missing backups or missing backup strategy, oom.
K8s health itself, fixes out of memory by restarting a pod, solves storage by shipping logs out and killing a pod in case it still runs full, has a rollout startegy, health checks and readiness probes.
It provides easy deployment mechanism out of the box, adding a domain is easy, certificates get renewed centrally and automatically.
Scaling is just a replica number and you have node Autoupgrade features build in.
K8s provides what people build manually out of the box, certified, open sourced and battle tested.
It is much easier to maintain when you look at those numbers.
Every VM had 4 cores and 20gig me.
Run on quite big blades
In most cloud environments a "core" can be just a thread.
Older machines have had 4-CPU socket based chassis with many cores as well. Definitely doable.
Alone the Paradigma shift from doing things step by step vs describing what you need and than things happen on it is a game changer.
K8s is probably 100x easier than Ansible.
And Ansible also has it's bigger ecosystem like Ansible tower.
Basically your k8s control plane but in bad
I've actually used both in conjunction and it was decent: Ansible for managing accounts, directories, installed packages (the stuff you might actually need to run containers and/or an orchestrator), essentially taking care of the "infrastructure" part for on-prem nodes, so that the actual workloads can then be launched as containers.
In that mode of work, there was very little imperative about Ansible, for example:
- name: Ensure we have a group
ansible.builtin.group:
name: somegroup
gid: 2000
state: present
- name: Ensure that we have a user that belongs to the group
ansible.builtin.user:
name: someuser
uid: 3000
shell: /bin/bash
groups: somegroup
append: yes
state: present
Deploying apps through Ansible directly can work, but most of the container orchestrators might admittedly be better suited for this, if you are okay with containerized workloads. There, they all shine: Docker Swarm, Hashicorp Nomad, Kubernetes (K3s is really great) and so on...
Using config management and introducing config drift and management of the underlying operating system is a lot more to think about, and a lot more that can go wrong.
We had all of these problems with self developed automatisation.
It still was garbage.
K8s just solves those issues out if the box.
- Only get servers with IPMI so you can remote reboot / power cycle them.
- Have said servers netboot so they always run the newest OS image.
- Make sure said OS image has a config that isn't broken so you don't get full inodes and so it cycles logs.
- Have the OS image include journalbeat to ship logs.
- Have your health checks trigger a recovery script that restarts or moves containers using one of a myriad of tools; monitoring isn't exactly a new discipline.
Yes, it means you have to have a build process for OS images. Yes, it means you need to pick a monitoring system. And yes, it means you need to decide a scheduling policy.
I wrote an orchestrator pre-K8S that was fewer LOC than the yaml config for my home test K8S cluster. Writing a custom orchestrator is often not hard, depending on your workload, - writing a generic one is.
K8S provides one opinionated version of what people build manually, and when it's a good fit, it's great. When it isn't, I all to often see people spend more time trying to figure out how to make it work for them than it would've taken them to do it from scratch.
My experience still counts for something and the example with those 700 VMS is something I didn't just saw once.
If I understand it right: VMS were not there because people needed VMS they were there because people needed compute.
We moved everything to k8s and we were able to do this because k8s can
For me it's now much more about proper platform engineering and giving teams more flexibility again knowing that the k8s platform is significantly more stable than what I have ever seen before.