Helm (the Kubernetes package manager) 3.0.0 has been released | Hacker News Reader

Helm (the Kubernetes package manager) 3.0.0 has been released

helm.sh

142 pointsmfer61 comments

Helm is problematic in my view.

To new Kubernetes users, Helm appears to be the "Official" package manager, and the "right way to do it".

It's not.

Many helm charts are out of date, have subtle bugs, and are not actively maintained. Upstream projects often have their own way of customizing and rendering Manifests.

The biggest flaw in Helm is the decision to use text-templating for semantic data. This leads to ridiculous situations like having to specify the number of spaces to indent a block, so the output YAML can be parsed properly.

Promising alternatives are:

* Jsonnet [0]

See kube-prometheus [1] for a complex project that uses Jsonnet to accomplish things that would be much more difficult using pre-defined templating.

* Cue [2]

from the creator of BCL/GCL (Borg/Google Config Lang), which is what Jsonnet is essentially. He's written quite a bit about Cue and how he intends it to address the shortcomings of Jsonnet.

[0] https://jsonnet.org/articles/kubernetes.html [1] https://github.com/coreos/kube-prometheus [2] https://cuelang.org/docs/about/ [3]

Two things to consider...

First, there has been a lot of talk about how much work it is to onboard to Kubernetes. There have even been recent discussions in Kubernetes SIGs. In UX terms, Kubernetes has a high barrier to entry.

I realize that many people who use and know Kubernetes like it. We folks are a group that has gotten past the high barrier to entry. Most have not.

Second, Helm is a package manager. It's like apt, yum, homebrew, and others. It lets someone who knows how to run an application (ops business logic) on a platform (like debian or k8s) and package it up so someone else who doesn't know this can easily run an application. This is why so many use brew and apt to install something rather than install everything by hand.

Something like jsonnet doesn't solve for these situations.

I know some folks don't like text-templating. I'm not a big fan of go templating. But, templating is something that people are used to and generally comfortable with. This means it has a fairly low barrier to entry.

Notice, a lot of the reasons for why Helm does things is to reduce friction for a large audience. To reduce the barrier to entry.

Yeah, the core reason to use Helm is for the ecosystem. The value that comes from being able to quickly set up and compose packages on https://github.com/helm/charts/ is larger than the (very) annoying YAML templating and chart reliability issues.

Lots of Helm charts have bugs, but lots of PyPI/NPM/Maven packages have bugs as well.

Jsonnet, Cue, and operators are nice, but unless someone can create an ecosystem that is as valuable as the Helm ecosystem, I don't see Helm disappearing.

This resonated with my recent experience. I have been using Docker and docker-compose quite comfortably for a few years now. I thought it'd try and dip toes into Kubernetes to extend my containerization skill-set and learn something new.

I ended up shelving it until I have time to take a proper course and dedicate the time to it. Docker compose was very easy to learn for me, with a few easy early "wins" and reasonably comprehensive and helpful documentation when I wanted to expand into new areas. For some reason, I really struggled grokking how to structure a k8 chart. Maybe I'll try again after a good night's sleep.

Have you tried plain manifests? Unless you need to support a lot of permutations (because of many environments, or you're distributing it and need to support many deployment options), Charts are generally a suboptimal way to get things running in k8s.

The major "wins" for using charts are usually around being able to use the same set of manifests with different values interpolated in. Which in turn, is really only valuable if you're supporting a large number of permutations that need to be kept in sync.

Thanks very much for the suggestion! I really didn't know where to start and the first few sites seemed to dive right into charts but I'll check out manifests this time. Thanks again.

I think all of this is indicative of a deeper problem, which is YAML is a great config but doesn't work when you need code.

When you're doing simpler deployments then config files are great. It keeps the learning curve flat for new users. The problem is that as users get more experienced, they ramp up the complexity, then start trying to shoehorn more abstraction into a format where it doesn't fit.

The most visible example is the need for code reuse. Any sufficiently complex artifact is going to have a lot of similar sub-components. What you want is a macro or subroutine that translates simple parameters into common patterns. What you end up with is a lot of brittle copy-pasting.

Text templating is a kludge that usually works fine until it doesn't. At the end of the day everything ends up messily converging to Greenspun's Tenth Law.

I agree, a package manager is designed for installing a graph of versioned (effectively immutable) artifacts like a binary or code package. Kubernetes is built to run services, which are not static but continuously running and react to inputs. Services need monitoring and self-healing to ensure they are healthy and operational.

In the future I predict we are moving towards a model where we have ways expressively define configuration (CUE) locally which we push to Kubernetes operators [0] which do the heavy lifting to deploy, monitor, and manage/upgrade a defined service that the operator is responsible for.

[0] https://kubernetes.io/docs/concepts/extend-kubernetes/operat...

> The biggest flaw in Helm is the decision to use text-templating for semantic data. This leads to ridiculous situations like having to specify the number of spaces to indent a block, so the output YAML can be parsed properly.

My mind was blown when I saw this the first time. Let's hope that the developers are planning on supporting alternate templating schemes.

The complaints about Lisp parentheses sound especially funny in the context of that kind of mainstream DevOps tooling https://github.com/helm/charts/blob/eceb82f6308578b824d58f80... (EDIT: yep, I'm a Lisp fan and a Kubernetes guy at the same time, and this kind of templating is like worst kind of "blasphemy" for me :)

Same here, and with Helm 2 it wouldn't even return an error, just silently fail with a timeout... Spent a week debugging that...

I'm a fan of jkcfg[1]. No crappy text templating, no having to learn a new language.

1: https://jkcfg.github.io/#/

There is also https://dhall-lang.org/

The worst aspect of Helm is that is is not idempotent. If you "upgrade" a release, it always rewrites the config map that the release is stored in, even if there are no changes. The config maps also get corrupted if there is an interruption during a deploy. Ansible does an admirable job of being a better Helm. It's too bad there isn't a community for Ansible roles for use on Kubernetes instead of Helm charts IMO. It's more powerful and provides the same functionality with better behavior.

I highly suggest giving Helm 3 a try, as much of the release ledger system was overhauled. If you can provide links to the issues you're seeing, that's always helpful too.

Oh, I probably will give it a try. Helm is pretty much inescapable since the community has rallied around using it for distributing manifests. It's just sad to need yet another bespoke tool when the general purpose ones do a fine job.

Cue is intriguing. The flexible typing mechanism is very powerful. Alas, the documentation is lacking. There is quite a bit of musing around the type/value lattice semantic and how the core operation is "<set of nodes>: <constraints>". Then the first realistic example [0] introduces a whole bunch of new features that are neither explained, nor motivated.

* Templates: deployment <Name>: ....

* List comprehensions: [ "-\(k)=\(v)" for k, v in arg ].

* Conditionals: targetPort: Port if port != Port

* Special templates: expose port <N>: .... What does 'expose' mean? Doesn't look like folded struct.

* Some form of comprehension/conditional composition: for k, spec in deployment if len(spec.expose.port) > 0. Can we nest two comprehensions? Can we zip? Can we reduce?

* Side note: What are the debug capabilities? One major usability issue with GCL circa 2010 was that debugging required sifting through MB [or was it GB?] of intermediate struct expansions.

* Side note: I wish they used Python f-string syntax: "-{k}={v}" instead of "-\(k)=\(v)".

[0] https://github.com/cuelang/cue/blob/e5d8d09b3ba2e4f48c84a3b5...

I also don't like Helm for these and more reasons, and thus I often have the wrong personal incentives to point out that the yaml indentation issue can be solved in a trivial way: yaml 1.2 is (for all practical purposes) a superset of JSON, thus all you have to do when you emit a bit of yaml from a template is to pipe it through the "to json" function and the resulting string is valid YAML yet independent of the indentation of the insertion point.

Exactly right. I do the same when dealing with YAML in general.

I've been using Pulumi to actually do the templating and deployment together, which I think works quite well.

It supports Kubernetes, but also quite a lot more!

Helm was great when I first started to learn Kubernetes, but overall has been extremely annoying and I would not recommend it for anything. I personally no longer use it on any cluster and if a specific piece of software in a cluster is on Helm, I immediately feel dread since I have no idea how that chart is going to behave on an upgrade or a change

A recent example of Helm being a problem for me is their Elasticsearch chart. I tried to shutdown my data nodes to increase disk space on all of them, but I noticed after scaling down, the first instance was taking a _very_ long time to shutdown.

Turns out, there's a default lifecycle hook (pre-stop.sh) that drains the node and rebalances the shards to the other nodes as a pre-stop hook. (to me, this is a bad idea, the cluster operator should control draining and rebalancing the nodes.)

Guess what, there's no way to comment out the draining functionality since the shell script is mounted as a read-only ConfigMap. Annoyingly, the only solution was to disable re-balancing across the entire cluster, and then jumping on each node and run $ kill on the shell script for each node.

I've ran into strange defaults like that multiple times. I've ran into images randomly being changed, charts being upgraded but breaking your live deployment. Overall just a nightmare.

Kubernetes is actually quite understandable and consistent. Helm adds a layer of complexity that makes you scared to upgrade or change your cluster. If you stick to Kubernetes primitives like Deployments and StatefulSets, it's super easy to know what changes are going to occur in your cluster.

This is a big step forward for Helm. Tiller has been a major pain point in the past, although I understand the design decisions why it was placed here in the past.

> In Helm 3, we now use a three-way strategic merge patch. Helm considers the old manifest, its live state, and the new manifest when generating a patch.

I will have to test this out, but honestly I think that any sort of "strategic" merge is probably the wrong approach to take when dealing with so called "idempotent" infrastructure.

Strategy is how you deal with idempotency and consistency in a distributed environment.

Your manifest is idempotent but isn't applied instantly.

You can apply a manifest, and get into some broken state, how do you recover? What if you try to roll back and things get worse?

I suppose if you're not using git-ops related deployments, but Git has been proven to work well for this exact process. If you are tracking your releases as part of a Git commit, then you rollback with that.

Doesn't seem that complicated, but maybe I'm missing some benefits of this strategy that others with more experience managing K8s resources have had.

Edit: Retroactively speaking you also don't need to leverage etcd to store n*x your K8s resources in a secrets file where n is the potential rollback number, it's all stored in a Git repository.

To add context, when updating objects in place a strategic merge is how the Kubernetes docs tell you to do and they they provide a kubectl example [1]. If this is not the right way to do it I would suggest that conversation would be good to have in the Kubernetes project itself.

[1] https://kubernetes.io/docs/tasks/run-application/update-api-...

I am aware of that, and here is one the of the issues I have with the strategic patch strategy:

> The patch you did in the preceding exercise is called a strategic merge patch. Notice that the patch did not replace the containers list. Instead it added a new Container to the list. In other words, the list in the patch was merged with the existing list. This is not always what happens when you use a strategic merge patch on a list. In some cases, the list is replaced, not merged. [0]

This seems like a lot of cognitive overload. I understand there are some use cases for this but really, all I want is to have my K8s resources all tracked with a git commit reference and then that is what is deployed exactly.

[0] https://kubernetes.io/docs/tasks/run-application/update-api-...

This is not how things work in reality. Many last-mile objects are merged into your Kubernetes resources at the last minute. Service meshes inject sidecar containers into your deployments. The Kubernetes API can update a Service's virtual IP addresses which can change over time. All of these object updates need to be taken into consideration during an upgrade, or you risk disrupting resources running in production.

I'm not saying there isn't a use case for this, it's just that for the most part I prefer dumping what I have from an immutable git commit into the Kube API. If something needs to happen after the fact for governance or injecting sidecars, so be it but that is out of the scope of what I am deploying.

Then that's fine. If no updates were made to the resource since the last update, then the patch will not look at the live state.

I highly suggest re-reading the FAQ front-to-back on this subject. I spent a lot of time explaining the details on this subject. If you have any questions/concerns, we are always happy to discuss further on github.

https://helm.sh/docs/faq/#improved-upgrade-strategy-3-way-st...

To take a step back... we're discussing a personal opinion on how to manage things compared to what k8s has documented and what's happening with other tools like service meshes.

I respect what someone has as an opinion. I just would like others to know where the opinion differs from the docs and tools out there. People can make their own choices.

Tiller has indeed been a major pain point.

Do you know how v3 prevents race conditions though? I assume there is some sort of locking mechanism via custom resources?

tiller was, imho, a huge burden on even testing out helm. setting up the right permissions was such a pain in the ass.

To add some context...

Tiller was not part of Helm v1. Tiller came in when Helm (which was developed outside of the Kubernetes project) was merged with deployment manager (which was part of Kubernetes). This produced Helm v2 which was part of Kubernetes. Helm grew large enough to spin off into it's own CNCF project.

Tiller was created early enough in the project that Secrets didn't exist at the time. This was long before RBAC or even the workloads API.

Tiller could have been remade as a custom controller which is how many things are created today. But, many people who share their apps work at organizations that don't let them install CRDs or use models like that. So, keeping the Helm footprint to a level that let's them and most others use it, if they choose, was a reason to remove Tiller. It also simplified the codebase a great deal.

String templating of yaml files. Everything is global scope. No sane way of reusing parts of a template.

We use helm, but for the life of me, there are some pretty poor design choices in there.

Shameless plug - I’ve worked on a dsl tool [0] at my previous gig that addresses these because I was similarly unimpressed with helm

[0] - https://github.com/cruise-automation/isopod

You can reuse parts of templates. One method is to create templates that are not automatically turned into YAML files and reuse them as needed. There are docs on it at https://helm.sh/docs/topics/chart_template_guide/named_templ...

ah yes, this technique, with its lack of template inputs and way of (not) handling indentation is exactly what I mean.

I'm sorry for the confusion. Go templates have piping which is how this is expected to be handled. There are functions of nindent and indent that handle the indentation. Inputs can be passed in. In fact, in the chart created using `helm create` this happen. For example, if you do `helm create foo` you can find snippets that read `{{- include "foo.labels" . | nindent 4 }}`. This includes a template, passes in `.` (the scope of variables for that template), and passes the output to `nindent` to get the needed indentation.

This is just an example and it's using go templates way of doing things.

And that's exactly the problem. With better templating language you shouldn't need this "| nindent 4" expression at all.

I see, the problem is with the Go templating syntax.

So, more context...

Helm was designed to support more than one templating system. The problem was that the majority of people were ok with Go templates so no one contributed support for other systems.

Many projects don't even need cluster state, they could have just used a Makefile and some sed templating [0] (get off my lawn). Thankfully in these cases you can use helm offline to render the templates then just apply them but it's not great.

Thus I'm sour on Helm, I'm rooting for Kustomize. [1]

[0] https://github.com/kubernetes/kubernetes/blob/master/cluster...

[1] http://kustomize.io/

kustomize only covers some simple (but common) use cases. If you can use it, you should since its modifications are semantic. Generally I find it is good for handling known differences between a few environments like local and production. But it isn't capable of replacing helm.

The Helm team did a great job addressing community concerns. I love to see this type of engagement and commitment to users.

> After hearing how community members were using Helm in certain scenarios, we found that Tiller’s release management system did not need to rely upon an in-cluster operator to maintain state or act as a central hub for Helm release information. Instead, we could simply fetch information from the Kubernetes API server, render the Charts client-side, and store a record of the installation in Kubernetes.

Oh, so now you're doing the obvious and simple thing? FFS...

Helm Classic was introduced at the first KubeCon... Wayyy back in November 2015. Kubernetes 1.1.1 was released earlier that month, and 1.0.0 shipped only 4 months prior to that in July.

Back then, Kubernetes had no concept of a ConfigMap. ReplicationControllers were all the hype (remember those?). The Kubernetes API was changing rapidly. When Helm 2 was being built, we needed an abstraction layer from the Kubernetes API to allow ourselves some room to guarantee backwards compatibility. Tiller was created as that abstraction layer. It provided us with a layer where we could control the input (via gRPC), the output (also via gRPC), and provide some backwards compatibility guarantees to users. We're pretty proud of the fact that Helm has maintained a strong commitment to backwards compatibility since 2.0.0.

Over time, Kubernetes' API layer has become more stable. Helm 3 is our opportunity to refactor out some of those protective layers we put in 4 years ago. Tiller being one of them.

Hope this helps provide some context.

Everyone has avoided running tiller server for a while and ran it locally (tillerless) instead. It serves the same purpose and works around the huge potential security issues tiller causes when ran in the cluster.

My wish for 4.0 is to decouple the pieces of helm. It has always done too many things. Package management, templating, deployment life cycle management.

Helm seems good enough as a package manager. But it forces me to use one particular text templating system. Why not allow a package to generate manifests with any system that it likes? This can be done reproducibly with a docker image in the k8s cluster (generate the manifests and send the manifests back to helm to apply them).

Deployment lifecycle management is complicated, and I can see why it is being coupled to templating values, but I don't think that must be the case.

It is great to see the 3.0 release just getting rid of Tiller, that's a useful step in this direction.

What happened to Lua plugin support that was discussed early on for v3?

Looks as though they'll tack it on later:

https://github.com/helm/helm/issues/5084

Does beg the question what have they been doing all this time. V3 has been a long time in the making.

Removing a significant piece of the architecture was a large undertaking. There's many parts of the system that required a redesign. Throwing on yet another major piece of work (including developing a new Lua VM from the ground up) would've delayed the release well into next year. Many users just wanted Helm without Tiller.

To provide some perspective, over 80,000 lines of code was changed between 2.16.1 and 3.0.0, including test infrastructure, the architecture, documentation...

We started development on Helm 3 last year in March, shortly after the first Helm Summit in Portland. A complete redesign of the architecture in 20 months time seems pretty par for the course for a project of this size.

Hopefully this will help gather more adoption.

Helm has always confused me. It’s an abstraction that seems to make things better and worse at the same time.

It sure is a leaky abstraction [0]. I would argue that the benefit of it is when spinning up your first Kubernetes cluster, the many community charts created by default can be installed easily with a > helm install stable/mysql.

However, it fails when the reality of running a service in production hits and you need to dive into changing a Chart template, requiring knowledge of not just the Kube API but also more often than not ugly template-fu to get the changes that you want reflected.

I really just recommend using raw K8s resource files for almost everything unless you are managing hundreds of similar releases.

https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

Agree. We built a management layer to take a DSL that was essentially kube yaml without the boilerplate required for common stuff and piped it on through.

If you’re using Kube you need to understand Kube. Conveniences are great to accelerate common tasks but as you say at some point ya gotta get into the guts of it and tweak things.

Using basic K8S resource files requires a lot of duplication, when using Kustomize or Helm isn't supposed to be that difficult to create an abstraction so that you can adopt you templates for multiple environments/purposes and inputs. With library charts in Helm 3, this should help even more to create a list of standard services or dependencies within your template files that can be quickly adopted throughout the organization. Once you have a few templates, then they can be quickly adopted.

Helm is a package manager in the same line as apt, yum, homebrew, and others.

A package manager lets someone who has knowledge of how to run an application on a platform (e.g., postgresql on debian) package it up in a way that others who don't have this knowledge can run an application without needing to learn it.

That's why I can use `apt install` regularly without needing to know the business logic of running all those apps on my system.

Package managers let you compartmentalize and automate expertise.

That might have been the intent of helm, but it does not at all reflect the reality of using it in the experience of most people I've spoken with, especially pre-3.0.

I've lost track of how many times helm has misrepresented the result of upgrade operations, even returning success when it actually failed, and what should be routine upgrades often break in ways that require detailed knowledge of both kubernetes and helm to recover from.

Yes, conflicts and failures do sometimes happen with package managers, but with helm we've seen it happen with nearly every Chart at some point or another, from the simplest single-resource pipelines to widely used community projects.

Helm 3 will help, but it's still built around the same messy string-templating system that ensures Charts are fragile and difficult to understand.

It definitely will. I've been waiting on Helm v3 to consider really using Kubernetes for things. Helm v2 had it's own authentication scheme and now that's baked into Kubernetes, so Helm can leverage that. Basically, old Helm was trying to own too many things (which, at the time, made contextual sense to do).

the last few orgs using k8s that i interacted with used helm (how i learned about it).

what is the competing tech?

Customize[1] is the competing tech re. Helm.

In fact, helm has moved closer ideologically in this new release to what Kustomize is.

[1]https://github.com/kubernetes-sigs/kustomize

If you look at the template language in Helm, Sprig, and compare that to the options available in a kustomization.yaml file, you'll see that they are in fact very different.

Reply on news.ycombinator.com