We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?
The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything.
What I'd like is to generate serialized yaml files ... you know ... with code. Using some typed programming language, to be able to build any abstractions required for the job, and get some checks, compiler errors, and even ability to have asserts, unit-tests and so on. Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me.
K8s was born to solve the same problem Borg solved or simply design in such a way to not have that problem.
One of them is the config files hell.
Borgcfg files ranks #3 in terms of line count across all of Google. (Borgcfg language is similar to Jsonnet, helm)
In terms of the amount of power wielded buy a single CLI, `borgcfg`, this little program that has less than 50k lines of nontesting code, stands at a peak dwarfs almost anything else inside Google.
I declared in 2015 that k8s will eventually enjoy the same borgcfg problem. And I was advocating code driving application management approach as you suggested, not the configuration as code (behind the infrastructure as code facade).
Well, it seems I am still on the track to be on the correct camp...
The yaml monster in k8s is quite unpleasant. It's always been surprising to me how we somehow ended up with executable BUILD files but totally static deploy files.
https://github.com/cuelang/cue/issues/33#issuecomment-483615...
@mpvl is one of my most admired Googler. I promised to contribute to the Cue project, but haven't found time (typical startup employee situation, but it seems there is no trouble to find time surfing hacker news...).
It does away with the notion that you need serialized yaml files, although in its current incarnation it can produce them, and instead runs programs which generates collections of named (and hierarchically defined) resources which can then be "applied" as a batch.
It uses lazy values to generate a directed graph to determine the order for creating and tearing down resources as well as safely propagate configuration outputs and secrets.
Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](https://github.com/kubernetes-client/python) without much trouble. Personally I prefer editing them as JSON instead of python objects, but it isn't too bad.
Not the OP, but..
1. YAML string templating makes it very easy to get indentation and/or quotation wrong, and the error messages can easily end up pretty far from the actual errors. Structured data should be generated with structured templating.
2. "Values" aren't typechecked or cleaned.
3. Easy to end up in a state where a failed deploy leaves you with a mess to clean up by hand.
4. No good way to preview what a deploy will change.
5. Weird interactions when resources are edited manually (especially back in Helm 2, but still a thing).
6. No good way to migrate objects into a Helm chart without deleting and recreating them.
7. Tons of repetitive boilerplate in each chart to customize basic settings (like replica counts).
It's a typical Go solution, in all the wrong ways.
> Alpha Disclaimer: the --prune functionality is not yet complete. Do not use unless you are aware of what the current state is. See ⟨https://issues.k8s.io/34274⟩.
I haven't used it in anger, so I can't add any disclaimer or otherwise of my own.
kpt is the recent Google-ordained (AFAICT) solution to this problem, but is ot yet at v1.
You could also resolve this yourself by either:
* versioning with labels and deleting all resources with labels indicating older versions
* using helm _only_ in the last mile, for pruning
Helm 3 does offer a solution: a JSONSchema definition file for the values.
Which works ... in a very Helm-like fashion. Meaning: it's messy and awkward.
Helm's stated goal is to be something of a package manager for apps in k8s, but this is fundamentally unworkable as shown by... Helm. It's hard to describe just how unworkable this idea is.
Let's start with an example, you want to install an app (let's say Jira) and a DB backend of your choice, postgres or mysql.
The first step where this all falls down is, it may or may not support your preferred DB. Sure, Jira does, but does the chart?
Assuming it does support your preferred backend, maybe it depends on the chart for the db you picked. If it does, it's going to install it for you, hopefully with best practices, almost certainly not according to your corporate security policy. This is also a problem if say, you have a db you want to use already, prefer to use an operator for managing your db, or use a db outside k8s.
You got lucky, and it supports your DB just the way you want it. Next question, do you want an HA Jira? Often, this part is done so differently that HA Jira and Single host Jira are straight up different charts.
Do you want your DB to be HA? Unfortunately, the chart Jira chart author picked to depend on is the non-HA one. Guess you're out of luck.
Maybe you want to add redis caching? Nginx frontend/ingress? Want to terminate TLS at the app host and not ingress? How do you integrate it with your cert management system?
We haven't even looked at the config, where you have to do everything in a variables.yaml file which is never documented as well as the actual thing it's configuring on your behalf, and is not always done in a sensible manner.
Hopefully it's clear from this that as a user, helm isn't going to work for you, because just as there's no such thing as the average person, there's no such thing as an average deployment. Even a basic one is filled with one off variations for every user that a public chart needs to support.
As a developer, helm is unworkable because you're templating yaml inside yaml. This isn't too bad if you're just tweaking a few things on an otherwise plain chart, but a public chart, that naively hopes to support all the possible configurations? Your otherwise simple chart is now 5-10x longer from all the templating options. Have fun supporting it and adding the new features you'll inevitably need to add and support.
As a counterpoint to all this, kustomize gets a lot right. I don't mean that kustomize is perfect, or even good, but I've found like k8s itself, it understands that the problem space is complex and to try and hide that complexity leads to a lesser product that is more complex because of leaky abstractions.
Kustomize acts as a glue layer for your manifests, so instead of some giant morass of charts and dependencies none of which work for you, you're expected to find a suitable chart for each piece yourself and compose them with Kustomize.
Going through the same example again as a user:
Your vendor has provided a couple basic manifests for you to consume, maybe even only one, because they're expecting you to supply your own DB. Since they only need to supply the Jira part, instead of having an HA chart and a Single node manifest, They just give you one manifest with a stateful set. Or maybe they give you two. One as a deployment and one as a Stateful set. The stateful set might also have an upgrade policy configured for you.
Since the vendor punted on the DB, you can do whatever you like here. You'll have to supply a config map with your db config to the Jira deployment, but that's okay, it's easy to override the default one in the manifest with kustomize. You are now free to use a cloud managed DB, an operator managed one, or just pull the stock manifest for your preferred DB.
Want to terminate TLS in your app again? Easy enough, Cert Manager will provide the certs for you, and supply them as secrets, ready to consume in your app, you just need to tell it where to look.
So now you have all the parts of your Jira deployment configured just how you like, but they're all separate. Maybe you just edited the stock manifests to get your changes in the way you like. Dang, you've created a problem when the vendor updates the manifest, as now you need to merge in changes every time you update it. That seems like a huge hassle. What if you could just keep your changes in a patch and apply that to a stock manifest? Then it's easy to diff upgrades of the stock manifest and see what changed, and it's easy to see what you care about in your patch.
All of this seems like it's getting kind of unwieldy, maybe we can make it easier. We'll have a single kustomization.yaml, and it'll be really structured. In it, you can list all the paths to folders of manifests, or individual files, or git repos/branches/subpaths. We'll also specify a set of patches that look like partial manifests to apply to these base manifests to make it clear what is the base and what goes on top. Then finally, for common things like image versions and namespaces, we'll expose them directly so you don't need to patch everything. We can do that because we're using standard manifests that can be parsed and modified.
That is kustomize, and as awkward as it is, it's just trying to make it clear what you're applying, and what your one customization from stock is in a maintainable way. It does a better job of 'package management' by not managing packages. This is pretty similar to how Linux package managers work. If you install wordpress on your server, it's going to install php, but it might not install apache or nginx, as everyone wants something different. It's definitely not going to install a DB for you. You as the admin have to decide what you want and tell each part about the other.
One of the benefits as a new user of k8s is the ability to grab a helm chart to get me most of the way with something like ELK. I want to go the way of Kustomize but can't seem to find the same with it?
[1] https://www.elastic.co/blog/introducing-elastic-cloud-on-kub...
But Helm is just a package manager and not a software delivery platform like you ask.
I mean do you have the same expectations from a Deb or RPM package?
If I give you a deb package that "contains" Jira, won't you have the exact same concerns?
The actual spec for a Deployment/Daemon Set/Stateful Set/CRD is usually super straightforward once you get the Kubernetes terminology, and most of the issues I've had with Helm have boiled down to "oh they haven't parametised the one config I need to change"
I've actually been using Terraform to manage Kubernetes resources at my latest job, along with literally everything else. I don't even use alb-ingress-controller or external-dns or any of that. I just have Terraform make target groups from the service resources. It breaks a lot less.
I don't know why you wouldn't, we don't say 'I don't know if .deb/.rpm/PKGINFO/etc. is worth it, most packages are just a binary'.
For Helm to add LoC you'd have to all but not use it, and even then it'd be about 3LoC. Setup is no harder, with N kubectl applies becoming 1 helm install.
I also started using terraform (directly instead of just triggering helm) - are you using kubernetes tf resources directly though, or the helm provider?
Someone here talked about the joys of using the API with protobufs. Something I want to try out because it sounds clean.
The only place where I find it useful is when the config is provider by an untrusted party. But even then scheme, lua would probably fit the bill.
Not strong feelings about it though, maybe I'm wrong.
All I want is a library for my favorite language, where I can easily output bunch of `yaml` files to feed to k8s. I guess pulumi is kind of like that, but I never had a chance to try it.
It is also very hard to prevent people from using the escape hatch of the general purpose language to "fix a problem quickly", while introducing unpredictable side effects.
Scons (using Python) and Gradle (using Groovy / Kotlin) are 2 cases where I have seen this go very wrong, very quickly. I now strongly prefer Maven, even with its warts and XML verbosity.
On the other hand, configuration languages also often suck, the abomination that is HCL (e.g used in Terraform) comes to mind (especially in its early days). Turns out it is really hard to make a good language, and a bad configuration language is even worse than using a general purpose language.
I do like Dhall though: sufficiently powerful so you never have to repeat yourself, while also preventing you from doing fancy stuff you will later regret, all with a very readable syntax, and the type system helps to catch typos / mistakes early. I now use it to generate all config files for tools that consume json/yaml, and often for other text based config files as well (where you can use it as a powerful type-safe template engine)
Use Pulumi, it does exactly this.
It’s very simple and worked quite well for me.
Disclaimer: work at VMware but on a different team (it’s an OS tool built by some of our engineers, which we use internally)
Check out Pulumi (I am not affiliated with them in any way)
- State. I cannot express how useful having the desired and actual state of a system is. It makes it so much easier to determined what has changed in between terraform applies which in turn makes tracking down problems easier.
- Providers. Being able to chain resources together from other systems is very powerful. As an example, I can:
1) create a database with a random name 2) create a user/password for that database 3) save the data above as a kubernetes secret 4) pass that secret into my kubernetes container for the service to use.
And you can easily change database to be anything such as a storage bucket, dns entry, twilio account, stripe account, etc.
Now try and do the exact same thing with helm!
- Providers Part 2: You can write your own providers to talk to your OWN services. I have done this and written up some go code to talk to our internal APIs when setting up new customers. It is very easy to create your own terraform resources to do this.
There are more reasons but I have work to do :)
Kustomize, too, doesn't get at the root of all the needs I've found myself having.
A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommend kubecfg [0] as it's a tool that "just works" and is built at the perfect level of abstraction: "just give me something and I'll find all the kubernetes objects inside of it". It's remarkably simple and forgiving and it makes it possible to describe even the most complex application configs + kube configs in a single language. You can also take it a step further and describe your CI stages (pre-merge, release, builds, etc), kube yaml, and infrastructure provisioning (terraform) all in one language.
> What are the benefits of using helm?
The main benefit is redistribution. Since everyone uses helm it's easy to give someone a chart because they know what to do with it. That's pretty much the only upside I've found.
If YAML is the assembly language of k8s, Dhall looks like what you would write the compiler in.
DRY configs make me very happy.
I don't see how they could support something in parallel to Dhall, and still enable all the features Dhall provides.
I don't think OP meant Kubernetes is changing to Dhall, but that we'll see something like Dhall on top of Kubernetes YAML/JSON that catches on.
I like your analogy though. For us, the assembly language is using the k8s API directly in Golang. The “compiler” is the golang Starlark interpreter extended with our own config API, like you would implement in Dhall. It’s just in this case, you can implement it in Golang, which has much much better tooling than Dhall does. A typed compiler, debugger, IDE, unit tests.... so much easier to develop and maintain.
Redistribution is important (and harder with a simple setup like make + kustomize) but I think obscuring the resources you need to run an application (custom or otherwise) is actually a bad thing for everyone in the ecosystem. If you're going to run something in your cluster, intimate knowledge of the resources required, the ways it changes your system is important.
Essentially the config language implementation would be the same logic that a helm chart would do, but you’re writing the logic in Go versus a text templating engine. You can easily unit test parts and rely on the compiler to catch basic mistakes. Way better than templating YAML.
We also provide escape hatches so people can easily patch the resources before they get serialized to YAML. People can use that to customize our standard deployment config however they want.
So far this has worked very well and been extremely easy to maintain.
- I already use Helm to install open source software; there are already so many Charts available. So that makes it easy to also use Helm for my own apps (I use it just for templating though)
- Helm templating language is more flexible, but that can also bite you since it can become complex. With Kustomize, changing a single environment variable value in a Deployment is quite some work compared to Helm (I don’t like JSON patching)
Config maps, enough said. I have a CD pipeline that writes environment variables to files, Kustomize eats them up and creates a configmap, environment variables in the deployment reference the config map, done.
Kapp as a layer on top of Kustomize is showing some progress. Ultimately Kustomize is like every other resource bundled with k8s, it’s a building block for higher level tooling.
We are not sold on using Helm however for K8s in general and are currently evaluating Tanka for some of our monitoring infrastructure. One of our engineers wrote up some notes on that here if you are curious https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issu...
I use helm as little as possible. Usually only enough to template out a single file with the different bits of k8s infra stuck together with `---`.
If I was doing some relatively serious work on custom services with minimal needs to conform to best^H^H^H^Hpopular practices, my service's configuration would be a neatly packaged Java program (its libraries for K8s direct API access are good).
There are different approaches to handling ops tasks; I favor the ones that are rooted in writing executable code over the ones with yaml.
Some bright chappie will probably write a Common Lisp library derivation library and k8s will be obvious and simple with that interface, but I'm not that bright - all I can see is the potential.
I find it pretty tedious for authoring my own Yaml specs; any time you need to do some sort of composition you have to write templates, and then mess with indentation operators, because significant white space. Templating yaml is a pretty bad idea in my experience.
As of Helm3 the most egregious part (Tiller) is gone, so it’s more recommendable.
Scaffold lets you write (roughly) native k8s yaml files and layer them together. IDEs know the schema for this yaml at this point, so you get code completion on all your input files. This breaks in Helm. Scaffold is good for simple apps, and falls down a bit if you need to dynamically configure your specs (eg render a parameterized manifest for each review app). The programming model for scaffold is a bit hard to grok, based on my experience of ramping up a team on it. Not rocket science just a bit unintuitive at first.
Declarative paradigms get a bad rap but I really just believe its people who missing imperative/functional programmimg. Theres a reason the infra space has been using declarative syntax for years now
The main reason folks dislike helm are 2 fold: 1) using string templating on a space formatted language is just messy and ugly for no good reason. If you don’t do anything terribly complicated it’s not too bad but quickly can be pretty fragile and unreadable. 2) golang(which I like generally) but the templating is messy and adds to the hard to read later problem
Personally find helm’s opportunity for code reuse to be restrictive and folks just end up cut and pasting a lot of declarations all over which is ... so terrible. But the first bullet above is pretty a fundamental design flaw.
They are not comparable.
For example, with Helm you can rollback applications to previous versions