What does the Kubernetes configuration format offer over configuration management systems like Ansible, Salt, Puppet, Chef, etc?
What k8s brings to the table is a level of standardization. It's the difference between bringing some level of robotics to manual loading and unloading of classic cargo ships, vs. the fully automated containerized ports.
With k8s, you get structure where you can wrap individual program's idiosyncracies into a container that exposes standard interface. This standard interface allows you to then easily drop it into server, with various topologies, resources, networking etc. handled through common interfaces.
I said that for a long time before, but recently I got to understand just how much work k8s can "take away" when I foolishly said "eh, it's only one server, I will run this the classic way. Then I spent 5 days on something that could be handled within an hour on k8s, because k8s virtualized away HTTP reverse proxies, persistent storage, and load balancing in general.
Now I'm thinking of deploying k8s at home, not to learn, but because I know it's easier for me to deploy nextcloud, or an ebook catalog, or whatever, using k8s than by setting up more classical configuration management system and deal with inevitable drift over time.
I think the build, deploy, start and run-time split is an important aspect that gets overlooked quite a bit, and is critical to evaluating tools at this point. That is why we aren't still doing everything with Chef or Puppet. Whether we continue doing it with Kubernetes or Pulumi or something else matters a bit less.
Repeatability is not the goal, as others in this thread have implied. The goal is trusting that the button will work when you push it. That if it doesn't work, you can fix it, or find someone who can. Doing that without repeatability is pretty damned hard, certainly, but there are ways to chase repeatability without ever arriving at the actual goal.
can't you do that just with containers?
And then one day I decided to set up kubernetes as a learning experiment. There is definitely some learning curve about making sure I understood what deployment, or replicaset or service or pod or ingress was, and how to properly set them up for my environment. But now that I have that, adding a new app to my cluster, and making it accessible is super low effort. i have previous yaml files to base my new app's config on.
It feels like the only reason not to use it would be learning curve and initial setup... but after I overcame the curve, it's been a much better experience than trying to orchestrate containers by hand.
Perhaps this is all doable without kubernetes, and there is a learning curve, but it's far from the complicated nightmare beast everyone makes it out to be (from the user side, maybe from the implementation details side)
It would mean I removed ~20% of the things that were annoying me and left 80% still to solve, while kubernetes goes 80% for me with the remaining 20% being mostly "assembly these blocks".
Plus, a huge plus of k8s for me was that it abstracted away horrible interfaces and behaviours of docker daemon and docker cli.
Are we talking about k8 base on your own server rack at your house?
Kubernetes is one way to deploy containers. Configuration systems like Ansible/Salt/Puppet/Chef/etc are another way to deploy containers.
Kubernetes also makes it possible to dynamically scale your workload. But so does Auto Scaling Groups (AWS terminology) and GCP/Azure equivalents.
The reality is that 99% of users don't actually need Kubernetes. It introduces a huge amount of complexity, overhead, and instability for no benefit in most cases. The tech industry is highly trend driven. There is a lot of cargo culting. People want to build their resumes. They like novelty. Many people incorrectly believe that Kubernetes is the way to deploy containers.
And they (and their employers) suffer for it. Most users would be far better off using boring statically deployed containers from a configuration management system. Auto-scaled when required. This can also be entirely infrastructure-as-code compliant.
Containers are the real magic. But somehow people confused Kubernetes as a replacement for Docker containers, when it was actually a replacement for Docker's orchestration framework: Docker Swarm.
In fact, Kubernetes is a very dangerous chainsaw that most people are using to whittle in their laps.
systemd-nspawn / machined makes the other systems look like very complicated solutions in search of a problem
Name may not be pretty but it's an official feature of systemd which is used to debug the systemd development and it is far easier to take backups incrementally because the container files are just plain files in /var/lib/machines/ and apparently you already have it if systemd is on your system. (May need an additional package to be installed from OS package repo.)
I run nspawn instances as development environments for developers and I can also run docker inside it.
So many people miss this. k8s is a very complex system and the talent it takes to manage it well, rare.
Extremely rare.
1st Question : Define k8s network, in detail, with all of the services and a set of services
IF you make it out of that one and the follow ups we can move on to the rest
So we end up with a plethora of full stack developers who can barely keep up with their current development stacks willfully deploying their software on systems that they're just barely competent with.
I know this because I almost deployed a side project with Kubernetes because it was expected of me despite the fact that being mediocre at it was the best that I could hope to become and that's an easy way to chop off a leg or three.
I've never used it but might find myself using it in the future.. can you elaborate on these a bit? I'm curious what the pitfalls might be
As far as I can tell: those are imperative. At least in some areas.
Kubernetes is declarative. You mention the end state and it just "figures it out". Mind you, with issues sometimes.
All abstractions leak. Note that k8s's adamance about declarative configuration can make you bend over backwards. Example: running a migration script post deploys. Or waiting for other services to start before starting your own. Etc.
I think in many ways, those compete with Terraform which is "declarative"-ish. There's very much a state file.
I would be somewhat surprised to find out Puppet and Chef weren't declarative either. Because setting up a system in an imperative fashion is ripe for trouble. You may as well use bash scripts at that point.
I've used Ansible for close to 10 years for hobby projects. And setting up my development environment. Give me a freshly installed Ubuntu laptop, and I can have my development environment 100% setup with a single command.
Sure, the steps are things like "if X hasn't been done yet, do it." That means it's idempotent imperative code. It doesn't mean it's declarative.
CFEngine is slightly less imperative, but when I was doing heavy CFEngine work I had a printout on my cubicle wall of the "normal ordering" because it was extremely relevant that CFEngine ran each step in a specific order and looped over that order until it converged, and I cared about things like whether a files promise or packages promise executed first so I could depend on one in the other.
Kubernetes - largely because it insists you use containers - doesn't have any concept of "steps". You tell it what you want your deployment to look like and it makes it happen. You simply do not have the ability to say, install this package, then edit this config file, then start this service, then start these five clients. It does make it harder to lift an existing design onto Kubernetes, but it means the result is much more robust. (For some of these things, you can use Dockerfiles, which are in fact imperative steps - but once a build has happened you use the image as an artifact. For other things, you're expected to write your systems so that the order between steps doesn't matter, which is quite a big thing to ask, but it is the only manageable way to automate large-scale deployments. On the flip side, it's overkill for scale-of-one tasks like setting up your development environment on a new laptop.)
The most simplistic task - execute some code in response to even in a bucket - makes kubernetes with all its sophisticated convergence capabilities completely useless. And even if somebody figures this out and puts the opensource project on github to do this on kubernetes - it just going to break at slightest load.
Not to mention all the work to run kubernetes at any acceptable level of security, or keep the cost down, do all patching, scaling, logging, upgrades... Oh, the configuration management itself for kubernetes? Ah sorry, I forgot, there are 17 great open-source projects exists :)
That's because you're not thinking web^Wcloud scale. To execute some code in response to event you need:
- several workers that will poll the source bucket for changes (of course you could've used existing notification mechanism like aws eventBridge, but that will couple you k8s to vendor-specific infra, so it kinda deminishes the point of k8s)
- distributed message bus with persistanse layer. Kafka will work nicely because they say so on Medium, even though it's not designed for this use case
- a bunch of stateless consumers for the events
- don't forget that you'll need to write processing code with concurrency in mind because you're actually executing it in truly destributed system at this point and you've made a poor choice for your messaging system
So once again, why developers need kubernetes for? If the most simple problem becomes a habitholy mess :)
For example, I can declare a Pod that mounts a Secret. If the Secret does not exist, the Pod won't start -- but once I create the Secret the pod will start without requiring further manual intervention.
What Kubernetes really is, under the hood, is a bunch of controllers that are constantly comparing the desired state of the world with the actual state, and taking action if the actual state does not match.
The configuration model exposed to users is declarative. The eventual consistency model means you don't need to tell it what order things need to be done.
This is basically the benefit of "containerization" - it's not the containers themselves, it's the constraints they place on the problem space.
Kubernetes gives you limited tools for doing things to container images beyond running a single command - you can run initContainers and health checks, but the model is generally that you start a container from an image, run a command, and exit the container when the command exits. If you want the service to respawn, the whole container respawns. If you want to upgrade it, you delete the container and make a new one, you don't upgrade it in place.
If you want to, say, run a three-node database cluster, an Ansible playbook is likely to go to each machine, configure some apt sources, install a package, copy some auth keys around, create some firewall rules, start up the first database in initialization mode if it's a new deployment, connect the rest of the databases, etc. You can't take this approach in Kubernetes. Your software comes in via a Docker image, which is generated from an imperative Dockerfile (or whatever tool you like), but that happens ahead of time, outside of your running infrastructure. You can't (or shouldn't, at least) download and install software when the container starts up.
You also can't control the order when the containers start up - each DB process must be capable of syncing up with whichever DB instances happen to be running when it starts up. You can have a "controller" (https://kubernetes.io/docs/concepts/architecture/controller/) if you want loops, but a controller isn't really set up to be fully imperative, either. It gets to say, I want to go from here to point B, but it doesn't get much control of the steps to get there. And it has to be able to account for things like one database server disappearing at a random time. It can tell Kubernetes how point B looks different from point A, but that's it.
And since Kubernetes only runs containers, and containers abstract over machines (physical or virtual), it gets to insist that every time it runs some command, it runs in a fresh container. You don't have to have any logic for, how do I handle running the database if a previous version of the database was installed. It's not - you build a new fresh Docker image, and you run the database command in a container from that image. If the command exits, the container goes away, and Kubernetes starts a new container with another attempt to run that command. It can do that because it's not managing systems you provide it, it's managing containers that it creates. If you need to incrementally migrate your data from DB version 1 to 1.1, you can start up some fresh containers running version 1.1, wait for the data to sync, and then shut down version 1 - no in-place upgrades like you'd be tempted to do on full machines.
And yeah, for databases, you need to keep track of persistent storage, but that's explicitly specified in your config. You don't have any problems with configuration drift (a serious problem with large-scale Ansible/CFEngine/etc.) because there's nothing that's unexpectedly stateful. Everything is fully determined by what's specified in the latest version of your manifest because there's no other input to the system beyond that.
Again, the tradeoff is this makes quite a few constraints on your system design. They're all constraints that are long-term better if you're running at a large enough scale, but it's not clear the benefits are worth it for very small projects. I prefer running three-node database clusters on stateful machines, for instance - but the stateless web applications on top can certainly live in Kubernetes, there's no sense caring about "oh we used to run a2enmod but our current playbook doesn't run a2dismod so half our machines have this module by mistake" or whatever.
I can buy a new laptop and be back to 100% in a few minutes. Though the amount of time I spent learning how to get there far exceeds any time savings. Ever.
A few years ago I even bothered to have two EFI loaders: one for amd an one intel, in case I want to change architecture as well.
Instead, every project I work on has a shell.nix in the root (and if it's not a project I control, I have a shell.nix mapping elsewhere).
Check it out, run nix-shell. Profit.
Once you're really ready for the big leagues, run it with --pure.
Back to your second question, you can configure the system through `/etc/nixos/configuration.nix` it is enough to configure system as a service. Pretty much everything you could do through Chef/Puppet/Saltstack/Ansible/CFEngine etc.
home-manager is taking it a step further and do this kind of configuration per user. It is actually written in a way that can be added to NixOS (or nix-darwin for OS X users) to integrate with the main configuration so then when you're declaring users you can also provide a configuration for each of them.
So it all depends what you want to do, the main configuration.nix is good enough if your machine to run specific service, that's pretty much all you need, you don't care about each user configuration in that scenario, you just create users and start services using them.
If you have a workstation, home-manager while not essential can be used to take care of setting up your individual user settings, stuff like dot-files (although it goes beyond that). The benefit of using home-manager is that most of what you configure in it should be reusable on OS X as well.
If you care about local development, you can use Nix to declare what is needed, for example[1]. This is especially awesome if you have direnv + lorri installed
you can add these to home-manager configuration:
programs.direnv.enable = true;
services.lorri.enable = true;
I also can't wait when Flakes[2] get merged. This will standardize setup like this and enable other possibilities.
Ansible uses YAML, but when I used it few times it felt that you still use it in imperative way.
The saltstack (which also uses YAML) was the closest from that group (never used CFengine, but the author wrote research paper and shown that declarative is the way to go, so I would imagine he would also implement it that way).
If you truly want a declarative approach designated from a ground up, then you should try Nix or NixOS.
But it has full power of ruby at your disposal (both at load/compile time and run time). So it usually turns imperative quickly.
But basic versions of these things are provided by Kubernetes natively and can be declared in a way that is divorced from configuring the underlying software. So you just learn how to configure these broader concepts as services or ingresses or network policies, etc, and don't worry about the underlying implementations. It's pretty nice actually.
Kubernetes isn't a silver bullet of course, there will be applications where running it in containers adds unnecessary complexity, and those are best run in a VM managed by a CM tool. I'd argue using k8s is safe default for deploying new applications going forward.
Unlike Ansible (and I suspect the others) where it's really only more of a 'run once' type of thing... And sometimes if you try running it a second time it won't even succeed.