Hacker News Reader

Comment by whalesalad | Hacker News Reader

Running kube is the most trivial part, especially with some of these tools like k3s/k0s.

Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days.

Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about.

I have not found a modern/open-source tool that is similar, but I have to imagine it exists.

View on HN

I agree that "apps" and deploying to the cluster is the hard part. Helm is definitely the most popular choice, but there are others.

kpt: https://github.com/GoogleContainerTools/kpt

kapp: https://carvel.dev/kapp/

cnab: https://cnab.io

There are many (many) more. But these are somewhat popular recently.

Thank you for the links. I had not heard of cnab until now but that looks really interesting. I will look into all of these.

I made a variation on the theme of kpt, you may find interesting: https://github.com/mkmik/knot8

(Rendered manpage at https://knot8.io/, if you like that exposition format)

> Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days.

kubecfg (with jsonnet) or CUE. Gives you actual composability and a powerful programming language for reducing boilerplate. Oh, and no text templating.

With kubecfg deploying/updating is:

    kubecfg update foo.jsonnet

No need to first build any Helm images or for any scaffolding. Just get `kubecfg` on your PATH and you're done. Version things the same way you version the rest of your code. Build your own abstractions that solve your particular problems as you go, instead of relying on someone's idea on how your deployments should work. Perfect for building orga-wide microservice deployment boilerplate.

Helm is horrible. Please use Kustomize. Life will get better, it did for me. I also use Sops (with goabout/kustomize-sopssecretgenerator) and ArgoCD for a very slick well integrated system. The CRDs in ArgoCD are so well done that it feels like native K8s CD.

Big fan of ArgoCD here. The development team is also very responsive with good processes in place, from my experience in the repo.

Kustomize and Helm take different routes. Kustomize expects the user to know Kubernetes resources and their schemas. Helm and charts target the chart creator being being an expert in Kubernetes while the chart consumer (the Helm user) doesn't need to be an expert. They can have a simple interface to deploy an application.

One of the issues for those running applications in Kubernetes is the complexity. Kubernetes is complex. So much so that people like Kelsey Hightower have said Kubernetes is a platform for creating platforms. For most people to learn Kubernetes will slow down their velocity which is a negative.

Most people need a simpler system on top of Kubernetes to make them effective.

Helm itself is pretty awful through.

For instance, when people upgraded to 1.16, where Kubernetes retired apps/v1beta1, Kubernetes automatically transparently converts these types.

Helm, having once created these resources as v1beta1 will completely refuse to touch the deployment. Without some specialized extra tooling that people created in the wake of this, resolving this requires deleting the entire namespace and reinstalling the chart. Issue on helm repo closed as wontfix.

This still comes down to assumptions.

If someone was using raw kubernetes manifests instead of Helm and charts they would need to know the schemas and details of those manifests. Then, when 1.16 came out they would need to update to the new apiVersion and modify their manifests for any differences in schema.

This requires someone who knows and understands k8s resources. This IS NOT a typical app dev or person who wants to run apps in k8s. I've been in numerous circles of app folks who have complained about this expectation from the k8s community.

On the main Helm project they have the stance that what's in a chart is up to the charts authors. Just like what's in a debian package is what's up to the debian package authors. That those folks should do a good job maintaining them. That's why it was a wontfix issue.

What we see as an underlying issue with k8s is the experience that app devs and people who want to run apps have to deal with. It's currently like expecting an app dev to write in assembly and know when assembly codes are deprecated or changed. It's in the pre-high level language phase. This comparison to assembly is one that's come out of the k8s community itself.

I generally do not disagree that the experience for appdevs on k8s is bad, but I also disagree that they should be the audience to begin with.

Helm is still functionally broken. A helm chart that was kept up to date still could've used v1beta1 early on. All it takes is a cluster that hasn't been patched in a while. This is 100% a helm issue. (Note that this bug also triggers if the chart has been updated, but helm recorded a different GVK)

What you actually need is a platform team that provides a middle layer. I was part of such a team between 2018 and 2021, and my main takeaway was to not let developers have write access at all after slicing a namespace with 25k pods out of etcd because gRPC refused to operate on it at that point (that was a misconfigured CronJob).

You need something like KubeVela, not Helm.

https://kubevela.io/

There is no need to use Kubernetes if it slows down your velocity, just use docker or direct drive some VMs. If you NEED Kubernetes then you simply won't get most of the benefits it provides by learning some short cut tools. Tools like helm are easy to use, but do so at the cost of breaking core design elements and a massive increase in complexity just below the surface.

Here's the rule of thumb for Kubernetes use tools to manage the files not the cluster, the cluster description is the source of truth not the cluster.

For some more complex deployments (i.e. not simple batch jobs or web services), we've used the Kubernetes API directly from programming languages like Python or Go. Often as a controller running within the cluster. That allows us the full data structures toolset and freedom of a programming language.

It reminds me of the old article "Beating the Averages": http://www.paulgraham.com/avg.html

Afraid to admit I had not even considered that idea, good thinking.

Yeah, just writing some code that talks to the Kubernetes API really ought to be more common.

It's not a great API, but it certainly beats writing bash scripts that shell out to kubectl and yq.

In the same vein, I wish people knew that YAML isn't even really a native Kubernetes thing, it's just one of the many ways you can serialize k8s objects...

In my experience, engineers with a formal education in Data Structures and Algorithms tend "get it" in a way that others don't- that the underlying data structure is a set of maps/structs and YAML, JSON, etc. are just text formats.

> writing bash scripts that shell out to kubectl and yq

We did this for a while, from a container rendering out our stack in Go, when server-side apply was not a reality yet.

The main issue is that lifecycle management of resources beyond writing code for a single deployment is hard to do with the typed clients, and the dynamic client used to be even more of a mess than it is now. There's a lot of implicit magic happening, baked into kubectl in a non-reusable way you just couldn't replicate before 1.16.

When we started back in 1.13 we wrote a function that took a set of structs as an arg and ran a kubectl subprocess to deploy it. We still had all of the niceness of native data structures. Later we transitioned to more native API calls.

> Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about.

The lack of this is what frustrates me so much about all of these things. Writing some bundle of yaml but just telling the management software "here go run this 'app'" with no state or version management of it makes me feel like I'm running a stack like it's a windows desktop. Just random shit everywhere and I don't know what any of it is. Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows.

All feels like a huge step backwards to me. Let me use version control as a source of truth, not just a spot to throw things in.

"Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows."

See Argo CD.

This is how I feel too.

At my workplace, we can easily spin up a kube cluster locally but our Helm charts aren't transportable for local development. They can be modified, torn apart, extracted from, but we don't have a seamless and elegant way to say "Hey, this thing in production, that's running via Helm/k8s, I want the same thing locally ... with these environment variables adjusted"

I don't think that's a difficult thing to accomplish and there is appetite from others I work with to do so, but there's _effort_ behind it.

Maybe there are smarter ways and we just don't know it yet.

Have you heard of Flux? It's not a manifest abstraction tool per se (like Helm or Kustomize), but automates the management of your manifests using GitOps. It sounds like it might be what you're looking for.

https://fluxcd.io/docs/concepts/

We are using tanka [0] and are failry happy with it. The jsonnet language allows nice layer of abstraction while still allowing to directly generate kubernetes resources when necessary. We wrote in-house library to map (very) simplified .jsonnet file to kuberenetes deployment; after all, 90% deployements are mostly same and differs in minor details only. Seems to work reasonably well.

0: https://github.com/grafana/tanka

In previous project, we had Kotlin DSL based definitions, and even though I only read it, it seemed to make a lot of sense: static typing, and reuse of code e.g., between QA and production environment definitions.

I think it may have been this: https://github.com/fkorotkov/k8s-kotlin-dsl

I've found kustomize to be really nice. Much less powerful than Helm, easy to learn, good enough for most tasks.

https://github.com/kubernetes-sigs/kustomize

jsonnet, works great and is composable however you like

Reply on news.ycombinator.com