Hacker News Reader

Comment by stevefan1999 | Hacker News Reader

stevefan1999

I really want something that is Docker Compose but for Kubernetes. I mean that I can have a simple way to declaring resources in just like Docker Compose, but I run the environment in Kubernetes so that I can get to test the behaviors when there are multiple copies of the softwares running together. I do rely on Kubernetes heavily for distributed and networked software deployment, so it is even better if we can emulate things like latency or burstable packet loss so that we can do a controlled chaos test for reliability test. I tried Skaffold, Tilt, Devspaces and Devpod/Coder v2, none of them are really simple like Compose.

View on HN

This is a problem we, as a company, have thought about a lot, but we always concluded that Kubernetes is already the simplest abstraction of a distributed system that is feasible for the diverse needs that the biggest companies out there have.

We previously built a package manager for Kubernetes to abstract it in the simplest way possible `glasskube install app` but we failed because every abstraction needs to follow a "convention over configuration" pattern at some point. Also, we weren't able to monetize a package manager.

With Distr (https://github.com/distr-sh/distr), we have actually been able to help companies not only package but distribute and either manage or give their customers a way to self-manage applications. Our customers are able to land on-premises contracts at enterprises way faster than before, which is also a clear ROI for paying for Distr.

So, I don't think that you can get the flexibility of a distributed application orchestrator with a simple declarative YAML file if your target environments are diverse.

Same, I've tried three or four times to make it work, including one attempt that just translated compose.yaml into k8s yaml, and every time I came away thinking, "just use k8s". K8s yaml looks complex, and can start to feel very boilerplate, but attempts to hide the complexity often just lead to something not-flexible-enough because it encodes convention over configuration, and inevitably some project runs into limitations and pretty soon you've just built an abstraction layer that leaks or is equally complex/verbose and now you have to learn something new.

Just use k8s and follow similar patterns is the conclusion I've arrived at personally.

Helm mostly does that. Not a huge fan of a text templating engine generating yaml but once you get your chart setup with a few variable inputs, you can continue using it for a bunch of other stuff with minimal new config.

The inputs (values) are yaml so you can make it look exactly like a Docker Compose file if you want (wouldn't be surprised if there's some charts floating around that do that)

I've recently been dipping my toes into k8s / kustomize / helm, and I recently had a situation where having a base deployment yml template that I wanted to reuse across various deployments. I had a look at Helm and I was frankly shocked how bad the templating was with Go templates, it was close to unreadable and felt very brittle!

I did that too, and ended up just skipping helm and using envsubst to interpolate the values I need at runtime from env vars. Nearly everyone preferred that approach. YMMV of course.

BTW, this is what Flux V2 (GitOps) supports as well and we do at our company. It's worked well and covered almost all our use cases.

I think that's a good approach if it works for your use case. Sometimes you might want something slightly more sophisticated like basic logic (loops/conditionals). In those cases, you can still use helm but you have an extremely simple template and avoid many of the "can't read this template" helm pitfalls.

How so? I find it much the same to other templating engines like Jinja, though I'm definitely not a fan of the syntax. But that hardly matters anymore with LLMs.

I think it's a combination of text templating + yaml (which is whitespace delimited). When templating html with Jinja, it's not a big deal if the indentation isn't right. In helm, best case you get a syntax error, worst case you end up overwriting keys and producing something syntactically but not logically correct.

Specifically with YAML, Go templating requires some special operators to get anything useful done and it ends up being an unreadable mess.

Yeah, that's fair. I don't think it's as bad if you make your own charts and can more liberally hardcode things. Community charts tend to have an insane amount of "knobs" so you can basically change everything being templated.

I don't know if I'd necessarily call it brittle, though. You can use `helm template` and various linters to validate the generated yaml is correct (and use something like pre-commit to autorun)

Shared up the chain, but bringing your attention to it as well: https://nixidy.dev/ solves all of these issues.

Have you looked into some of the Docker Compose Bridge transformers? We’re using it to scaffold out k8s apps for UDS from compose files https://github.com/defenseunicorns-labs/compose-bridge-uds/

This is a great idea for some vibe coding tbh and you can customize it to match exactly the resources that you need as everyone has slightly different requirements for their k8s stack.

The simplest way to start probably would just be to create a custom Helm chart and define what you want in `values.yaml`.

You should take a look at https://nixidy.dev/.

Reply on news.ycombinator.com