Deploying Kubernetes with Ansible | Hacker News Reader

Deploying Kubernetes with Ansible

erichreich.com

55 pointskdubb57 comments

My conclusion is that using Ansible with Kubernetes is the WRONG way to go about it. So now instead of a directory of yaml or helm charts we have a bunch of ansible uselessly wrapping the yaml for application.

The problem is when you're in a situation where "over here we need YAML document A, and over there we need YAML document B." You either shove a templating system in front of your YAML and fight with that or throw in the towel just use a programming language where constructs in your YAML map to objects.

I think we need to burn this whole thing down and rethink it at this point. A huge amount of the total operational cost of our business is working around weird YAML problems and helm charts.

Had one a while back where quoted strings containing padded numbers in were used for something. "01" fine ... "05" fine, "06" fine, "07" fine, "08" kaboom. 01-07 were treated as strings and 08 was suddenly being treated as octal, which it's not and exploded.

Helm is a pile of garbage. k8s speaks JSON just fine as expected.

Try Tanka + Jsonnet to restore some of your sanity, also allows you to keep using Helm during your transition to a better place.

This is just the same problem in JSON, another loose schema’ed mess. Got my fair share of problems with that at the moment as well.

If you feel really strongly about getting type safety then there is Pulumi + Typescript though I really hate how it tries to be a "better" kubectl apply and fails miserably.

Definitely room for someone to make something better than Helm but I think you should give Jsonnet a shot if you haven't already.

> tries to be a "better" kubectl apply and fails

How does it try? How does it fail?

I don't see any interactive CLI commands that could be seen as a replacement for kubectl.

It's not about the commands. `pulumi up` is what you are looking for and the k8s resources that accompany it.

kubectl apply, especially with server side apply is effective just a mechanism to push the declarative definition of your k8s objects to the server.

Pulumi instead tries to manage the lifecycle of each of those objects itself when it really has no business doing so. This results in a lot of roundtrips to the Pulumi state store (which is implemented horrendously inefficiently btw) and generally takes forever and a day compared to kubectl apply.

Most of this is down to an impedance mismatch between the models themselves. Pulumi + k8s works but it's just not a great experience. I say this with an absolute -ton- of experience with both options.

This can be turned off by setting an annotation on the resources.

I'm aware but it's not really ergonomic.

For instance Pulumi also generates random names for resources, again you can override these but again, not very ergonomic to work around etc.

IMO tools like Tanka + Jsonnet, or Cue are just better fit for k8s because you can stick with server side apply which is a much more native experience.

Pulumi also supports server side apply

https://www.pulumi.com/registry/packages/kubernetes/how-to-g...

In what way is the kubernetes API loose schema'd?

It’s not loose schema at all. All the tools that wrap it are.

You don’t know something is going to work unless you actually do it.

Ah gotcha.

I wonder if anyone's converted the API spec into json-schema, might be easier to plug into tools then.

edit: yes, and the tooling in front of it. https://github.com/yannh/kubeconform

following on original work from https://github.com/instrumenta/kubernetes-json-schema

Helm3 made it a solved problem years ago.

Gosh, I wish the author wouldn't sound so arrogant presenting me his reinvented wheel.

What your Ansible is doing (looping over the inventory, imperatively executing tasks) is reinventing what a Kubernetes-internal Operator is doing after you asked it to reconcile your changes. Thing is: You're not only reinventing the internal thing, you will eventually recreate internal logic, e.g. some internal dependencies and knowledge "what to do in which order". Good luck with further developments on the Kubernetes upstream. Not cool.

Kubernetes does what it can, to give the user a declarative DX. You might not want to fight this. kustomize and some _trivial_ scripting is usually all you'll need. If you need more, consider writing custom Operators.

> kustomize and some _trivial_ scripting is usually all you'll need.

So not just k8s alone, k8s + kustomize, no Helm charts, right?

or can you do k8s + Helm, no kustomize?

or can you do kustomize to deploy Helm charts which abstracts away and translates to underlying k8s API calls/generated YAML?

Hm take it with a grain of salt, but I'm not really a fan of Helm. It might make sense in some bigger corps where you have something like "Platform" teams with dozens of consuming teams to consolidate and version some kind of service. But I lack trust and frequently same-mindedness with Helm chart authors on how to deploy and configure certain things such that I'll use my own thingy anyway.

I forgot to mention the CD tools we have, e.g. Flux or ArgoCD. Use customize to render a "final" yaml for your env/version/whatever and throw it at Flux/Argo (e.g. using GitOps) and be happy. That's imho the Kubernetes way of life.

couldn't agree more, but we are "stuck" in managed ansible + k8s

although i think kustomize wouldn't make a nicer picture than ansible, but push vs pull .. i prefer pull

My preference is to use kustomize with argocd applicationsets.

If helm is how some application is distributed we just render the template with our values file and store that in git, even though argocd can also use helm charts. We'd just prefer to see the diff in git instead of argocd so the deployment is still automated.

Yes

I thought this was about deploying a Kubernetes cluster with Ansible, but nope, it's really about deploying YAML files to Kubernetes, with Ansible.

Just...Why?

When all you have is a hammer, everything looks like a nail.

> Just...Why?

I can imagine some cases where you might want to setup infrastructure-related software on your cluster and to do so through Ansible (using manifests or Helm charts under the hood), so that you have one template that you can use for setting up clusters later, for various use cases. For example, you might want a custom ingress controller, or to deploy a service mesh like Istio, or maybe Kiali as well. Perhaps you want to launch Rancher on your cluster to give your engineers an arguably easier way to interact with the cluster, or to explore it in a more visual way than just using something like the kubectl CLI (though tools like k9s are great, too: https://k9scli.io/).

Edit: Many suggest that using something like Argo CD would be a better tool for this, due to its specialization, though.

I actually utilized a similar approach when working with Ansible + Docker Swarm (since I needed a more lightweight orchestrator that played nicely with Docker, but to manage the nodes with Ansible): I deployed my own "ingress" (just a web server container with some configuration), some supporting software like Sonatype Nexus for storing custom images, Portainer for graphic cluster management and some other software for similar concerns. Once that was done and other server configuration (e.g. user accounts, groups, permissions, folders, services etc.; though most less relevant for the containers themselves) were set up, other CI/CD pipelines could deploy apps across the cluster without Ansible being involved - but it was good for setting up the base for it all.

But if you wanted to use Ansible for deploying almost all of your apps, like in the article? I guess you could do that if you have a lot of buy-in into using Ansible and just prefer the way you do things with it. Admittedly, using Helm or one of the alternatives in this case might be a better choice.

Sounds like the title should be "Deploying To Kubernetes With Ansible", since this isn't about deploying a Kubernetes cluster?

... oh god why.

Also thank you for the very apt and red flagging summary kind sir.

Yeah, title made me think kube-spray

I had the same impression.

I think the lesson people need to start taking is that any form of declarative form of configuration will spawn an ad-hoc ecosystem to turn it imperative. The complexity of situations that can be expressed is by a configuration system is proportional to the complexity of that system. Simple and declarative is an oxymoron for anything but simple cases. Imperative systems can express arbitrary complexity and you will need it. It's why troposphere, terraform-cdk, pulumi, a billion templatating systems over k8s yaml files, and Ansible exist.

The lesson is that shockingly few people have realized is that the problem space is not, nor has it ever been, served by declarative solutions. You can describe the state you want a system to be in but there will inevitably be procedures involved to do some of the work. It is unavoidable. At the point people began developing templating around their configuration, ah, templates... which themselves have imperative language constructs awkwardly (and badly, cough Golang) embedded in them, it should have been clear to everyone with a shred of talent that it wasn't a good solution.

Declarative configuration is wholly expressible by imperative languages. The converse is not true. Declarative configuration is appropriate for declaring state (i.e., values), and not for describing procedures.

Stop using half-baked YAML crap and start using real tools.

I think the root of the issue is not that we need imperative languages for configuring cluster workloads - declarative state as an API contract between clients of the cluster and the cluster itself is a good thing. The reconciliation model is what makes kubernetes work, and that should be preserved.

But I do think we need better tools to generate said declarative state, regardless of its serialization format (yaml, JSON or other). Hand editing the raw declarative state is clearly untenable, hence the development of the various templating solutions. But I don't see an issue with using an imperative program to generate the declarative state either, as long as the declarative state is what forms the substance of the API to the cluster.

Imperative isn't really unavoidable, all imperative programs can be expressed as functional declarative programs which is much more natural to something that should be declarative in nature (I don't want to know everything you did to get/maintain your network.)

The problem is that people want to express functions as extensions to declarations in yaml, XML, etc, instead of in a functional language that is heavy on declaration.

Do you have any examples of deploying to Kubernetes with something you consider real tools? I agree with what you are saying but have found very few real world companies using real languages for this.

At work we use typescript to generate yaml files which are then synced to their respective clusters via Gitops. It works fairly well. All cluster/namespace differences are encoded into our typescript code base, so we can take advantage of a real programming language to generate cluster specific yaml all sharing a common “base”.

Pulumi, cdk8s.

I would suggest to look at something like Argo CD, Flux or Gitlab CI/CD's Kubernetes support instead.

Wat? If you really need to, you can use Ansible to bootstrap a cluster but I wouldn't use it for anything more than that - that's the responsibility of CD. Ideally you would just have Terraform apply a manifest that installs Argo or Flux and that's it.

Ansible is the regular expression of devops, you can use it almost anywhere to try and solve a problem, but now you have 2 problems.

Ansible makes you comfortable with mutable infra, but with enough time I always regret using it.

This went a different direction than I'd expected. That is, configuring your cluster resources rather than the host machines themselves.

I managed to grab 5 or so Raspberry Pi 4s before the availability issues. I've been (slowly) getting a home Kubernetes cluster running over the last year or two, and Ansible helps immensely; I:

* Write Ubuntu 20.04 (or now 22.04) to MicroSD card.

* Configure the initial user + SSH-PK, hostname, and network via config added to the SD

* Boot up the new Pi

* Run Ansible playbook to get it ready for k3s (lightweight but "full" k8s); including bind-mounting all high use paths for containerd/logs/etc to a USB NVME-SSD, adding boot args to the Pi's GRUB, etc.

* Push all my bashrc/tmux/vim/etc configs for consistent environment if I ssh in.

* Use Ansible ad-hoc for misc changes or package updates.

In terms of the "deploying to Kubernetes", I've recently got FluxCD working and love it. I think "GitOps" is the way to go in terms of clear reproducible config.

I have GitLab running on a local Synology (NAS) which kicks off any container builds and hosts the Git and container registry. The FluxCD operator on the cluster polls my "cluster as code" repo and syncs state. Each project defines a Helm chart and Flux pulls the chart from each repo and applies any variables (as configured in the "cluster repo") or "kustomizations".

Flux let's you setup "dependency" relationships for components. So I have it add the sealed-secrets operator early in, and put PK-encrypted secrets in directly the Git repo for it to"unpack" in the cluster.

That being said, the author's Anisble-driven approach is a much quicker way get something on a cluster ASAP, versus setting up the initial Flux "pull from cluster" system.

I'm primarily a software engineer, so this was also largely an exercise in getting more comfortable with the operations side of things.

Just thought I'd share my experience since only so many people let me rant about my FluxCD/k3s/GitLab home dev environment.

`kubectl apply -f myshit/` is a faster way of getting things on your cluster, ansible is just a bunch of wrapper on top of that.

And when you outgrow it you can move on to `kubectl apply -k`...

Until you need to update it, or apply it to multiple environments, or any number of tasks where the concepts of configuration parameters and dependencies become important.

Edit: I'm not saying Ansible is an appropriate substitute (it absolutely isn't), rather that `kubectl -f` is not a scalable solution.

All I'm saying is ansible is essentially that and no it isn't at all scalable.

Some of the other controllers in the Flux family are also nice. There's also the Terraform and Helm controllers and they all work in a consistent way and produce consistent metrics about what they're doing

Not really the best technical solution for this challenge. And this is from a big Ansible + Packer person that deploys 10Ks of VMs and infrastructure with Ansible.

10,000+ VMs? What do you work on?

The art of ansible is knowing when not to use it. I'm guessing that ansible to K8s is an example of when not to do so, but I'm no expert.

I started with Ansible and k8s and.its.fine.

Not really hard, you get it up and running fast enough (who ever wants to setup a k8s cluster like this has the week to do so).

But honestly: don't!

I switched to rke2. Alternativly try out gardener from sap or anything else between Ansible and managed.

You do not want to take care of bare metal k8s.

And rke2 is still not perfect but it's rancher support will be better and better and more than just you are using it.

But be real: if you don't need air gap, go to a cloud provider.

Ah and btw open nebula also started supporting k8s by deploying rke2!

I don't think you read the article. It's about deploying resources onto k8s with ansible, not deploying or maintaining k8s.

Huh? Than that's even worse.

Use argocd or flux

Does the author ever heard about using loops in Ansible? with_dict can save you around 20-30 lines of code.

Use argocd! It's the best tool for the job.

Can and should be configured as IaC.

Argocd can manage itself!

Personally I prefer Flux, but YMMV.

Flux is a much better gitops option.

As your cluster grows, so does the complexity of managing it. That's where Ansible comes in.

I've never worked anywhere sizeable that uses ansible to manage kubernetes. My one brush with it was at a small shop and it caused more problems that it solved. It was just a couple of guys that happened to use ansible for everything else, right or wrong, it seemed like a lot of "i've got a hammer..."

In the pre-managed kubernetes world I did see an ansible stack for standing up a cluster that seemed ok, but once you get past setting up a base cluster ready to take 'customer' workloads ansible is just not a great tool.

I completely agree that for managing the host nodes/machines, Ansible is a huge win. Though this article focuses on deploying ON the cluster.

Ideally the complexity of deploying ON a cluster (i.e. 'kubectl apply') doesn't increase with its size. In some ways it actually gets simpler, as you can worry less about per-node scheduling due to resource constraints/packing.

Reply on news.ycombinator.com