Hacker News Reader

> kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml}

This is fine for the first deploy, but if you delete a resource from your manifests then kubectl doesn’t try to delete it from the cluster, so in practice you need something like Terraform (or perhaps ArgoCD) which actually tracks state. And of course as you mention, you probably want to generate these manifests to DRY up your YAML so you can actually maintain the thing.

I would love to hear from others how they solve these problems.

> as the whole idea of text templating YAML is... thoroughly ignorant

1000%

You can use -prune flag to apply to delete resources removed from your configs.

Which is still in alpha

Before there was Helm, I wrote my own tool and framework in Ruby (called Matsuri) to generate and manage manifests. I still use it. However, the source of truth is still on the kube api server. I did write in functionality to diff what is in the git repo and what is in the source of truth. There is an additional abstraction that bundles together resources so that a single command can converge them all together. The underlying mechanism is still kubectl, and the code merely generates the manifests to send it to kubectl via stdin.

I did not think about a “delete”, and I usually don’t want it to be automatically deleted. But that’s a great idea for that convergence mechanism if I have something that explicitly adds a deleted resource line to make sure it stays deleted until otherwise specified.

The Ruby code can access anything Ruby to generate the manifests, so I have a Turing-complete language and I can use class inheritance, mixins, method overrides. I was also able to write a suite of helpers to take a raw yaml manifest from an upstream project, and use a functional programming style to transform the manifest to something else. If I ever have time, I’d write the ability to import templates from Helm.

This was written for a one-person devops role, intended to be able to manage almost-similar things across different environments (dev, staging, prod), which works great for small, early stage startups, for our level of complexity. At this level, I don’t need it to track state.

When our team size grows, I’ll have to think up of some other patterns to make it easier for multiple people to work on this.

The other thing is that for much more complex things, I would just write operator code using Elixir, and bootstrap that into k8s with the Ruby framework.

> Before there was Helm, I wrote my own tool and framework in Ruby (called Matsuri) to generate and manage manifests.

I've built these kinds of things too in Python (or Starlark). It's an improvement over YAML, but I often feel the pain of dynamic typing.

> The Ruby code can access anything Ruby to generate the manifests, so I have a Turing-complete language and I can use class inheritance, mixins, method overrides.

I actually don't want any of these things in a dynamic configuration language. I kind of just want something that can evaluate expressions (including functions, objects, arrays, scalars, and pattern matching). If it's Turing complete that's fine but unnecessary. I explicitly don't want I/O (not in your list) or inheritance, mixins, method overrides, etc--unnecessary complexity.

Dhall probably comes the closest, but it has a tragic syntax and I'm not a good enough salesman to persuade real software teams to learn it. I've also heard good things about Cue, but my experiences playing with it have only been painful (steep learning curve, don't really see the upsides).

I don't really see this tooling as a dynamic configuration language so much as a manifest generator. That means I also have tooling for diffing, debugging, and converging manifests. If you are just focused on configuration, I can see why Turing-completeness will seem like unnessary complexity.

I use classes and mixins to be able to generate similar manifests with slight differences across different clusters or environments. I sometimes use imports (I/O) for manifests provided by an upstream (such as from AWS docs), do transforms to get manifest I want.

I modelled the design off of Chef, which will also declaritively define and converge a set of systems towards desired state. Well-known paths and conventions helps keep things organized.

I havn't had a problem with dynamic typing, but I have also been using Ruby in application development for over 10 years. You might see it as unneeded complexity, but I have been able to use the flexibility for years now. This tooling was designed for a team that uses Ruby as the primary language, and uses designs and idom that would be familiar for a Ruby dev team. It is definitely opinionated and I don't expect it to be universally useful for everyone.

I've been using the following command to great effect:

kubectl apply --kustomize manifests/ --prune --selector app=myapp

It cleans up old stuff from the cluster and also allows you to split your manifests across multiple files.

Whoa. I never knew about --prune. <mind-blown emoji>

Yeah... I'm not really sure where this idea of not being able to use the tools available out of the box to deploy apps and do networking comes from. I deploy YAML. If I feel like my YAML is too big, I DRY it using kustomize. I can use --prune if I'm worried about stuff sticking around in the cluster. For networking, I... don't do anything? We get DNS built in. Just use the service name. What else is there to do?

Label selectors are hard. They might require knowledge as advanced as high-school geometry to understand. The ability to draw a Venn diagram isn't free, you know!

External DNS, certificate management, and a whole bunch of other stuff if you're not using a cloud provider's managed Kubernetes (e.g., network attached storage, load balancers, ingress controller, etc).

> kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} > > One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is).

Agreed 99%, but there is one useful thing that Helm provides over Kubectl+structured templating (Nix/dhall/whatever): pruning objects that are no longer defined in your manifest.

However, that's solvable without taking on all of Helm's complexity. For example, Thruster[0] (disclaimer: an old prototype of mine) provides a pruning variant of kubectl apply.

[0]: https://gitlab.com/teozkr/thruster

I really struggle with seeing the value add of helm. I use it, but mostly with a "less is more" approach. It is semi handy in a dev/staging/prod sort of environment, but really not all that much.

What I don't get is some people in my company thought it was a good idea to make a "docker helm chart"... That is, a helm chart capable of deploying arbitrary containers, ingresses, etc... Like, a true WTF, since the values file ends up looking awfully similar to a k8s manifest :D.

> the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is

Could you expand on that? It sounds like an interesting position (bordering on the philosophical), but I don't know enough about Kubernetes to gauge its accuracy for myself.

k8s is built for extension with validation of the extensions you add ( https://kubernetes.io/docs/concepts/extend-kubernetes/api-ex... )

Helm is just sort of dumb text manipulation with a TINY bit of deployment management built on top of it. There isn't really a whole lot that helm buys over extending k8s.

There's two things:

1) Text templating YAML is just bad. It's the serialized format of some structured data - instead of templating its text representation (and dealing with quoting, nindent, stringly-typed variables, and general YAML badness), just manipulate the structures directly before serializing them. For example, write some Python code that composes these structures in memory and then just serializes them to YAML before passing it over to kubectl. You can also use ready made tooling that uses domain-specific languages well suited to manipulating and emitting such structured configuration, like jsonnet/kubecfg (or CUE/Dhall/Nix combined with kubectl apply -f). Eschewing text templating is not only more productive by letting you build abstractions (eg. to deal with Kuberenetes' verbosity with labels/selectors, etc.), it also allows you to actually compose more complex deployments together, like 'this wordpress library uses this mariadb library' or 'this mariadb library can take a list of objects that will be upserted as users on startup'.

2) Those YAML/JSON manifests aren't even that native to Kubernetes. A lot of things go behind the scenes to actually upsert a manifest, as Kubernetes' resource/object model isn't nearly as straightforward as 'apply this YAML document full of stuff' would indicate (there's state to mix in, API/schema changes, optional/default fields, changes/annotations by other systems...). With k8s' Server-Side-Apply this can now be fairly transparent and you can pretend this is the case, but earlier tooling definitely had to be smarter in order to apply changes. Things like doing structure-level diffs between the previously applied intent and the current intent and the current state in order to build a set of mutations to apply. What this means is that Helm's entire 'serialized as YAML, manipulated at text level' stage is not only harmful and a pain in the neck to work with (see 1.) but also unnecessary (as 'flat YAML file' isn't any kind of canonical, ready-to-use representation that Helm had to use).

That is a very helpful summary. I generally favor declarative configuration (like CloudFormation) and I think it's mostly due to my work being infrastructure focused: "I need VPC, LB's, ASG with those immutable Images, a queue, buckets etc". But in my recent work with a customer where the infrastructure is "EKS with datastores, and Gitlab CI" .. most of the complexity is people creating abstractions on top of abstractions of stuff in helm (and also .gitlab-ci.yaml with tons of severy nested includes). And in this case the text templated yaml is really painful. Something that would be like CDK for k8s could actually be amazingly useful. Lots to ponder, thank you.

The tool that you want is probably Jsonnet. Take a look at Tanka or one of the other Jsonnet options. There is also Dhaal and a few others but Jsonnet has a lot more traction.

> Something that would be like CDK for k8s could actually be amazingly useful.

Such thing surely does exist: https://cdk8s.io. And here's the relevant announcement blog post: https://aws.amazon.com/blogs/containers/introducing-cdk-for-....

> Could you expand on that? It sounds like an interesting position (bordering on the philosophical)

I'll make the philosophical case: text-level templating of a computer-readable format is almost always the wrong approach. It becomes unnecessarily hard for someone who later needs to read the code to ensure that it generates valid (both syntax and semantics) markup. It's also more hard for tooling to verify the validity, because the syntactic validity may depend on inputs to the program.

Compare the approaches of PHP and JSX. They both accomplish a similar goal (interleaved HTML and control flow), but JSX works at the element tree level and makes it impossible to generate poorly-formed HTML syntax -- it becomes a compile-time error (though it doesn't ensure semantics, e.g. a tbody outside of a table is legal). Compare with PHP, which very much allows you to generate invalid HTML, because it's just a text templating language.

(From what I can tell, Helm works more like PHP; if I'm wrong my philosophical position stands but might not apply here)

I’m a big fan of kapp[1] for this sort of thing, it’s basically the same as kubectl apply, but with an app name and state tracking .

    kapp deploy -a my-app -f ./examples/simple-app-example/config-1.yml

It was created by folks at my employer but I use it because I just like it’s simplicity.

[1] https://carvel.dev/kapp/

> in fact, you probably shouldn't be using Helm

Funny that the thing you shouldn't use is what the entire Kubernetes ecosystem uses for deployment. It's almost like there's no good way to do it.

Except it's... Not what the entire ecosystem uses. Yes, it's popular because it made for "app store"/"package repository" style operation, but I have yet to actually use it for longer than a few weeks despite running k8s projects across multiple companies since 2016.