Hacker News Reader

Comment by mac-chaffee | Hacker News Reader

The Kubernetes API is itself a leaky abstraction over many different concepts. Putting docker-compose over top of that is just an even leakier abstraction.

Even their example of using Ingresses is already leaky because there is no similar concept of Ingresses in docker-compose, so they're hacking it with labels. What if I need to set a higher request timeout for my app? I have to set a docker-compose label that converts into an Ingress annotation (the leaky part of the Ingress abstraction)? And this is somehow less cognitive load?

Like all similar attempts to simplify deploying apps, it only works if all your apps fit in a nice box. Anything slightly non-standard (like needing different classes of storage volumes) will expose the leakiness. Even Kubernetes kinda still expects all your apps to be stateless web services and will fight you otherwise.

View on HN

I thought maybe I was just getting old and crusty, but this really does capture my primary complaint with K8s - everything that isn't standard, is so special and unrelated to the underlying technology that you need to relearn how K8's interacts with every layer of abstraction just to have half a chance of it working.

> everything that isn't standard

Could you give an example? I thought k8s and its YAML/practices was the standard?

I've always felt the standards are more about organization and common practices, whereas the implementation is still the wild west. Unfortunately in my experience, abstraction, even if structured, doesn't solve the main issue: it turns out the engineering operation of infra is still pretty hard. At least you know where to look to find the chaos though.

> whereas the implementation is still the wild west.

in my own experience, i would've thought "ok, let me aptinstall k8s and then apply some yaml to it so i can avoid going to the cloud for a massive k8s bill for almost no reason"

standing up kubeadm on a single master + worker node setup on a VM (like a droplet or EC2) is basically like, extremely non-standard + frowned upon

then, does anybody actually write pure k8s yaml? or do they write jsonnet/kustomize/helm charts? do they apply with kubectl or the helm CLI or the argocd CLI?

> then, does anybody actually write pure k8s yaml?

You can think of plain k8s yaml as a sort of machine code for k8s. You _can_ write yaml by hand, but it’s not generally how things are done outside of initially learning how k8s works.

The tools you’ve mentioned all make working with k8s resources a lot more pleasant. Something simple like parameterizing the name of a secret instead of hardcoding it in N different yaml files saves a lot of time if you ever need to refactor, and being able to provide those values via a “standard” CLI tool makes automated deployments a lot easier compared to hacking together some yq commands. helm in particular tracks the history of any charts you’ve installed, and has a very simple “helm rollback” command which can get you back to the last working version of your application if things end up going wrong.

The exact tools used by each team will differ; you could use any of them. But no one in my experience is writing plain yaml manifests and kubectl applying them in production contexts.

He means part of the k8s standard. Once you get to the non-standard, cloud specific configuration, things go haywire.

For example try to configure a udp connection in k8s on aws.

oh, like azure k8s vs aws k8s vs gcp k8s? each has their own way to like... attach a volume for block storage or something? i thought that was all abstracted away/standardized?

> Once you get to the non-standard, cloud specific configuration

what's a common use case for when this is hit?

YAML practices… YAML is not a language, it’s a syntax tree. It pretends to be configuration and what it really becomes is some kind of a gimped template engine which people put layers and layers of code generators on top of to manage the complexity. XML was better and I’m willing to die on this hill.

Schemas and includes are great features, but you will have to defend the syntax hill on your own - imo the syntax is neither human nor machine friendly

Oh don’t get me wrong I don’t like xml. I just don’t like yaml so much more.

Yea, gotts agree on this one. XML hits the sweat spot.

Use the service resource in Kubernetes to order a load balancer in different clouds. You need different tags, they function and are configured differently, and you still need to learn how it works in your cloud.

Contrast this with the contract* K8 tries to sell you, which is "describe the intent and we'll figure it out for you" and you'll realize the cloud providers bunged up this whole thing.

That's not to say K8 is perfect, but if you think for even a second that wiring up resources outside your kubernetes cluster will be lower-friction, kubernetes is falling short of it's goals.

Yep. And the end result is a hodge podge of labels that make up an API that is much worse than the original interface, and requires significant amount of bounds/error checking in order to provide useful error messages until the codebase for it is no longer simple.

How would you make it less leaky then?

Helm charts are pretty standard, git/kube-native, and lead much more nicely into the more sophisticated operator/controller model.

One part I haven't quite put a handle on is where/when to use CRDs in either helm or OLM. The community still seems split on how to use CRDs.

I just give developers access to k8s and teach them about it. That's really just my admission that the Kubernetes API isn't perfect, but it's better than many alternatives. If I were to build something to hide k8s from developers, eventually I'd have to add so many extra options that I'd probably end up re-inventing the Kubernetes API anyway.

That may just be unique to my job. Maybe another company could force all apps into nice boxes that prevents their bespoke platform API from leaking.

Reply on news.ycombinator.com