I moved to docker swarm and love it. It's so much easier, straight forward, automatic ingress network and failover were all working out of the box.
I'll stay with swarm for now
I've had decent luck in the past with the K3s distribution, which is a bit cut down (but certified) Kubernetes: https://k3s.io/ Others might also mention K0s, MicroK8s or others - there's lots of options there.
It also integrates nicely with Portainer (aside from occasional Traefik ingress weirdness sometimes), which I already use for Swarm and would suggest to anyone that wants a nice web based UI for either of the technologies: https://www.portainer.io/
But even so, I still run Docker Swarm for most of my private stuff as well and it's a breeze. For my needs, it has just the right amount of abstractions: stacks with services and replicas that use networks and can have some storage in the form of volumes or bind mounts. Configuration in the form of environment variables and/or mounted files (or secrets), some deployment constraints and dependencies sometimes, some health checks and restart policies, as well as resource limits.
If I need a mail server, then I just have a container that binds to the ports (even low port numbers) that I need and configure it. If I need a web server, then I can just run Apache/Nginx/Caddy and use more or less 1:1 configuration files that I'd use when setting up either outside of containers, but with the added benefit of being able to refer to other apps by their service names (or aliases, if they have underscores in the names, which sometimes isn't liked).
At a certain scale, it's dead simple to use - no need for PVs and PVCs, no need for Ingress and Service abstractions, or lots and lots of templating that Helm charts would have (although those are nice in other ways). Unfortunately, Swarm isn't awfully good for job prospects, which I have to admit. Then again, one can sort of say the same about Hashicorp Nomad, yet it's also alive and kicking.
I’m honestly curious about Kubernetes and I want to learn more but wow, it really is hard to grok.
What does it mean that k3s is “certified”? Should I care? What’s the tradeoff?
When you mentioned Traefik my eyes glazed over.
Is there a picture of how all this fits together? One not created by a consultant selling their services?
The biggest thing to understand about Kubernetes is that you cannot understand it as a normal piece of software. You don’t directly control it, but rather, define the conditions under which a collection of automations respond to things going on. This is how it self heals. The whole system resembles more of a garden, where you are providing the conditions in which the plant grows, but you are not directly causing the plant to grow.
For example, if you understand a pod, then the next step is understanding how to keep a pod running, how to detect failures, what go do if it does fail, and how to change out one set of pods for the next. This would be a deployment.
You build your understanding by understanding more independent automations. For example, after understanding a deployment, you can look at how horizontal pod autoscalers work. An HPA knows nothing about a deployment and its internal state, yet will inform the deployment how many pods it should be running based on the metrics it sets. The HPA and the deployment scheduler otherwise work independently, and you can indeed write something that functions as an alternate HPA, and Kubernetes will do it.
There’s also an addon for cluster autoscaling. The cluster autoscaling knows nothing of deployments or HPA. It only checks pods that are being requested, but cannot be scheduled due to constraints. It then add nodes so that pods can be scheduled.
These three different processes operate independently, and don’t share state. They each watch for a limited set of signals and take limited set of actions. If any one of these processes goes down, the system can still limp along. Together, they create a much more complex behavior out of much simpler behaviors.
I’m sure it isn’t and I changed the emphasis to make that line more about the slope of the learning curve.
Thanks for this explanation. The inability to grok the internals is really hanging me up. I’ll embrace the simpler “distributions” as a way to understand the higher level constructs.
Does it also make sense to ignore the fact that this is built on Linux’s containers and namespaces?
Based on my very limited understanding Kubernetes is basically a “cloud operating system”. This is why the underlying components like networking and container managers can be swapped out. But I still don’t grok how the subsystems fit together.
The Linux container/namespace is useful in the sense that operational characteristics matter, and many troubleshooting can end up going deeper and deeper in the stack. But as far as understanding how to _design_ things, knowing how the pure abstractions are composed together lets you be able to apply it to many systems, whether it is Kubernetes or something else.
I suggest reading through the first few chapters of Roy Fielding’s PhD dissertation. He is known better for REST, but his dissertation was actually about architectural elements, properties, and constraints, and then applying to hypermedia as an engine of state.
Once you know how to reason through computing architecture, you can pick apart any architecture, framework, or paradigm, and know how to work with it, and when you fight against it. You don’t grok a system by finding out how specific parts fit with each other, but rather, how things are consequences of the architectural elements, properties, and constraints. Then, the specific way things work (and don’t work!) together pops out.
And don’t be afraid to dive into the messiness behind an abstraction. Knowing and being willing to do that can set you apart in your career.
I think that maybe I should have linked directly to the K3s GitHub, which goes into detail about what it is, what the components are, as well as links to more information about testing: https://github.com/k3s-io/k3s/
Kubernetes is a large and complex system, but doing end to end tests aims to ensure that various implementations work correctly, they have pretty in depth docs on all of this, including conformance tests in particular: https://github.com/kubernetes/community/blob/master/contribu...
So, if a distribution passes all of these, then it's likely that it will give you fewer unpleasant surprises. Here's a list: https://www.cncf.io/certification/software-conformance/
> When you mentioned Traefik my eyes glazed over.
Well, you typically do need an ingress controller that will allow exposing services to the outside and direct network traffic.
Traefik is liked by some, less so by others, but generally works okay - except that its documentation (for K3s in particular) isn't as abundant as it would be when using Nginx. For example, if you want to use a custom SSL/TLS certificate (without ACME) for all domains by default, figuring that out might take a while.
That said, if you don't like it, technically you could swap it for something else, however I personally had a cluster hang once while doing that due to some cleanup action getting stuck.
There are plenty of options: https://kubernetes.io/docs/concepts/services-networking/ingr...
> Is there a picture of how all this fits together? One not created by a consultant selling their services?
I actually found their diagrams to be surprisingly sane: https://docs.k3s.io/architecture
However, do be warned that there are many different ways how to build clusters that might have significant differences.
K3s is just one of the Kubernetes distros - that attempt to provide an opinionated view on how to mix and match the components out there to end up with something that will let you use Kubernetes and have things actually work as expected at the end of the day.
A bit like how we have OS distros like Debian, Ubuntu, RHEL and so on.
I think they were saying that is akin to the state of kubernetes, where everything is too new to really judge it today. Let me make clear my bias before I get to my opinion: I'm a big fan of what kubernetes has accomplished with managing the deployment of applications. But I think comparing current day kubernetes to a restaurant's grand opening is naïve. Kubernetes is amazing at what it does, and for how highly flexible it is. But it is highly flexible because of how advanced it has become, which in turn requires more knowledgeable administrators. You can take off with Docker Swarm because... it's very simple. It doesn't aim to be flexible, it just aims to be the docker-compose of distributed systems. Which is fine if you don't need the flexibility that kubernetes offers anyway.
And while I do appreciate kubernetes for all of its strengths, going back towards the point of this OP, I do believe it's a bit of a security nightmare. For example, the kubernetes operator pattern often just requires this application to run in your cluster with a ClusterRole granting it read and potentially write levels of access to ConfigMaps and Secrets around your cluster. Often without much in the ways of a method of telling the kubernetes API server what namespaces it is allowed to view those objects in. Worth mentioning though that secrets management is something that Docker Swarm also handles fairly poorly, and honestly isn't much of a security regression from the normal "deploy a VM that has all your decrypted secrets sitting on the system" anyway.
I wanted to eat some food so I went to the meat packing factory, hence disappointed I went to a drive through and bought burger.