Accidental complexity, essential complexity, and Kubernetes | Hacker News Reader

Accidental complexity, essential complexity, and Kubernetes

driftingin.space

137 pointspaulgb75 comments

The article is pretty lite, though I agree with most of the points.

However, I think that, especially in the context context of Kubernetes, this part is completely wrong:

> Containers were a great advance, but much of their complexity falls into the “accidental” category. There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code.

Containers are not used in Kubernetes or other similar orchestrators because of their support for bundling dependencies - that is a bonus at best.

Instead, they are used because they are a standard way of using cgroups to tightly control what pieces of the system a process has access to, so that multiple processes running on the same system can't accidentally affect each other, and so that a process can't accidentally depend on the system it is running on (including things like open ports). These are key properties for a system that seeks to efficiently distribute workloads on a number of computers without having to understand the specifics of what workload it's running.

They are also used because container registries are a ready made secure Linux distribution-agnostic way of retrieving software and referring to it with a unique name. quay.io/python:3.7 will work on Ubuntu, SuSE, RedHat or any other base system, unlike relying on apt/yum/etc.

I think they mean different things to different people. From the problems I have faced with customer-controlled OS installations, the biggest thing that they offer is configuration isolation (or rather independence). I have seen some truly crazy shit done by customer's administrators and even crazier shit done by management software that they install. Taking away that autonomy is huge.

Absolutely, but the context of the article was specifically Kubernetes.

In the context of Kubernetes, what do containers have to do with cgroups? I get that container runtimes typically provide a nicer interface for setting them up, but Kubernetes doesn’t use those interfaces anyway. If it was just about cgroups, they could have just run a regular process in cgroups.

Containers are a construct created by employing cgroups + namespaces together. Namespaces provide isolations in terms of filesystem, network, users, processes, and probably more.

Yes, but those are features of Linux that containers use; containers don’t have exclusive use of them. My original point was:

I stand by that. We might just be arguing semantics about whether that unit of code would be called a “container” or not. I’m leaving open the possibility that it would be (say) a WebAssembly module or static binary.

A container can also give your application assurance that the file system looks a certain way. But you are right that you can get pretty far with bundling dependencies and assets and configuring all nontrivial paths with environment variables.

However, the choice of containers for Kubernetes is a pretty fundamental one. It simplifies distribution of deployables to a single workflow also for programming languages that don't offer monolithic deployables, to legacy application not originally intended for Kubernetes, or that don't follow the 12 factor app principles.

Containers are a temporary solution while we wait for everyone to realize self contained binaries with static linking are the real solution.

No, containers give you things that static binaries just don't. How do you specify the maximum allowed memory for a static binary? The ports it opens? The amount of CPU usage? The locations it will access from the host file system?

Also, how will you distribute this static binary? How do you check that the result of a download is the specified version, and the same that others are downloading? How will you specify its name and version?

By the time you have addressed all of these, you will have re-implemented the vast majority of what standard container definitions and registries do.

The orchestrator can still use cgroups for resource constraints and isolation. Or it could use virtualization - it would be an implementation detail. But devs would not have to build a container.

Binary distribution, versioning and checksumming shouldn't need to be coupled to a particular format.

Obviously docker solves a bunch of disparate problems. That's kind of the objection.

At that point it's not an implementation detail anymore and that's kind of the point.

Static binaries are great but they don't solve every problem and they do to have to.

The vast majority of usecases don't need kubernetes, that's fine too.

You don't have to bash kubernetes because you don't specifically need it for your one use case.

There are plenty of other perfectly valid reasons to bash that dogpile.

> At that point it's not an implementation detail anymore and that's kind of the point.

Not clear what is meant by this. Can you elaborate?

> You don't have to bash kubernetes because you don't specifically need it for your one use case.

It is the platform teams at my employer deploy to, and with good reason. I think we do need it for our use case. But there are ways it could be improved, and the ability to deploy a simple Linux binary feels like one.

AppArmor or SELinux?

These don't solve all those problems. Moreover, it is something already employed by container runtimes.

It's what we did in C in the 80s and 90s and all the young hipsters chose to ignore. This industry loves going in circles.

In the same vein, I also heard that the new data exchange hotness is grpc. Reading a little into that thing it sounds very much like the good, old... CORBA. So many wheels reinvented every decade, sigh.

Many of the reasons CORBA failed are the reasons that gRPC is succeeding. CORBA tried to solve the wrong problem at the height of the object orientation fad - it tried to provide a standard mechanism for remote “objects” across languages which have different object semantics. It was a doomed idea from the start.

As for the industry going in circles, it only looks that way if you’re not familiar with the technologies in question. The idea that statically linked binaries are an alternative to containers doesn’t make any sense. In fact, it’s not that uncommon for a container to contain nothing but a statically linked binary. People don’t do that for fun, they do it for the additional features and standardization it provides over a standalone binary.

Distributed objects had nothing to do with object semantics in a PL. They're a lot closer to what we would now refer to as the actor model.

> Distributed objects had nothing to do with object semantics in a PL.

In a way, that was the problem. In practice for CORBA, there was a problematic relationship between its object model and the object models of its host programming languages.

CORBA was supposed to be language independent, but that meant that native language objects and CORBA objects weren't the same thing. There was a semantic gap, analogous to the one between relational databases and object-oriented languages. In the latter case, you can use an ORM-like solution or even just handwritten data access objects to achieve a native language object layer for your data model. With CORBA, that wasn't really feasible - you had to embed CORBA object implementations in your language of choice, and program to that non-native object model. In some ways it's comparable to .NET, but that has the advantage of using a common underlying runtime across all its language implementations. CORBA didn't have that.

You mention the actor model, but what examples are there of successful cross-language actor implementations? The reason for this is exactly what I'm talking about. The practical consequences of a cross-language distributed object model are problematic.

The closest answers I'm aware of share a common runtime - there's Erlang and Elixir both on the Erlang BEAM runtime, and Akka with Java and Scala on the JVM, with variants for C# that I'm not too familiar with. In both cases, the common runtime determines the shared semantics of the object model, so you don't get the same kind of semantic gap.

It's not a coincidence that what's ended up becoming popular are data-only formats like JSON and service-oriented RPC mechanisms like gRPC or even REST - these don't conflict with native language structures in the same way. As I said, many of the reasons CORBA failed are the reasons that gRPC is more successful - the most critical one being that's CORBA's notion of "object" was the wrong level of abstraction for language-independent distributed computing.

The executable piece is nice, but not the whole picture. Configuration and, more importantly, isolation of the runtime, are also huge benefits that come with containers.

Static linking is even worse. Now you traded a blob containing gigabytes of software making it difficult to audit and maintain with a smaller and more opaque binary blob.

Research shows that containers on docker hub, flatpaks etc are full of unmaintained libraries with vulnerabilities in them.

So no, we need to go back to proper dependency management and shared libraries.

Sounds pretty much like containers are the solution to containers.

The best formulation I saw is "containers are a solution to the Rube Goldberg Machine development pattern".

Since dependency resolution problems are hard for many popular languages (Node, Python, Ruby), you are not going to enjoy sharing the installed packages between two apps. Same with shared libraries for compiled languages.

Otherwise something like Nix + cgroups-based runner had a chance to suffice.

That is only a real solution if everything is running on the same operating system in the same environment.

What is the point of Kubernetes at all in your situation.

Everything already is running on the same operating system with Kubernetes where it matters (kernel, runc/crun, etc). Containers are a band aid to wrap an app with additional files/libraries/whatever. In Go for instance, I can include all this stuff at compile time (even conditionally!).

I'd love to see Kubernetes be able to schedule executables directly without using containers by way of systemd nspawn or similar. You could have the "container feel" without the complexity of the tool chain required to build/deploy/run/validate containers.

The tool chain to build/deploy/run/validate is the whole point to containers. It's something that unifies the deployment model of applications, unless you want to commit to a single programming language ecosystem. And not every ecosystem has tools to create a monolithic deployment artifact.

Even if you have monolithic deployment artifacts: by the time you have solved all the use cases that the image distribution workflow cover, you have basically reverse-engineered it. If you have already done that, kudos to you, you indeed don't need that workflow. And you probably will find it very easy to containerise those artifacts should you ever need to.

After thinking about this for a few minutes, the author might be on the wrong track connecting this with containers.

But I think there's something really there about "environmental linting". I know deep in my bones I need write access to make log files, but I don't know how many times I've debugged systems lacking this permission.

I know the log path won't be known until runtime, I know the port can be specified at runtime, but I think there's a ton of room for improvement around a tool that says - hey, you're making this set of assumptions about your environment, and you should have these health checks or tests or whatever.

I agree with you about this is what containers give, but I think the author is really on to something about the dev tooling and environment warning about what sorts of permissions are needed to, like, work.

Why not both? “Works on my computer” was absolutely a problem that docker advertised itself as a solution for.

> Containers are not used in Kubernetes or other similar orchestrators because of their support for bundling dependencies - that is a bonus at best.

> Instead, they are used because they are a standard way of using cgroups to tightly control what pieces of the system a process has access to

Well, sure, you get both. But the point is that needing a post-build step just to get that is accidental complexity. I'm by no means a Java fan, but the way JVM gives jars as a compile target and lets you run them with some measure of isolation is an example of how things could be.

With Maven, Gradle etc you can output a Docker container from a single package command.

And I don't understand this idea of using code bundles as the deployment artefact.

They don't specify how the code should be run e.g. Java version, set of SSL certificates in your keystore etc.

Do you really want Kubernetes to have hundreds of options for each language ? Or is it better to just leave that up to the user.

I think the suggestion is to use native Linux binaries with static linking of any libraries and resources in the executable.

Java programs would need to be compiled as a standard ELF. I think GraalVM can do this.

Only some Java/Scala programs can be compiled as a single binary.

It's been a concept that has been around for years but not progressing to the point where it comes close to negating the need for the JVM in a Production environment.

In the meantime containers works today and is significantly more powerful and flexible.

The fallback scenario just embeds the interpreter inside of the binary. That still nets you a single binary, unless I'm missing something?

I just don't follow the argument. You can also use Java tooling to wrap up a container image. Why is it accidental complexity that we settled on a language agnostic target that also wraps up a lot of the process isolation metadata?

Java is an outlier here. For most languages the process of dockerizing a codebase involves learning a separate tool (Docker). As someone who knows Docker, I get the temptation to say "who cares, just write a few lines of Dockerfile", but after talking to a bunch of developers about our container-based tool, having to leave their toolchain to build an image is a bigger sticking point than you might think.

But this is begging the question. If the work was more integrated with the compiler, it would still need to be learned. If your compiler of choice spit out an deployable unit akin to an image you'd still need something akin to the Dockerfile to specify exposed ports and volumes and such, no?

The Dockerfile doesn't specify volumes, those are configured at runtime. In the case of Kubernetes, they're configured in the pod spec.

As for ports, EXPOSE in a Dockerfile is basically just documentation/metadata. In practice ports are exposed at runtime (if using docker) or by configuring services (if using Kubernetes), and these are unaffected by whether a port is exposed in the dockerfile.

IMHO this is how it should be -- if I'm running a containerized database or web server, I want to be able to specify the port that the rest of the world sees it as, I don't want the container creator to decide that.

while I admit I'm being a little loose in what would translate where in this hypothetical, none of it seems so easily cut out.

You're right that 'containers' are essentially a higher-level pattern but the main underlying building block is arguably kernel namespaces, not just cgroups.

> With that in mind, I want to talk about the sources of complexity in Kubernetes.

I don't care about the complexities in Kubernetes, I care about the complexities it (cl)aims to be solving. I reread the article a second time to see if I missed something, but the whole article just seems to be talking about the complexities within kubernetes and justifying those.

I still don't understand why every application must be able to scale to infinity and maintain 99.999% reliability. Not everyone is Google and I feel like we'd be a lot better if if the not-Google companies just recognized that instead of layering on accidental complexity in the form of infinite scaling for your service that gets all of 5,000 requests a month.

It's funny, this article mentions Fred Brook's "No Silver Bullet" and then proceeds to tout a solution developed for Google scale problems as the silver bullet to whatever it is Kubernetes is actually solving. Like what does it even mean to be using kubernetes if you only ever have 1 or 2 instances (pods?) running and deploy infrequently (which I suspect is the norm rather than the exception)?

I have been using Kubernetes for my side projects and personal home services for almost 5 years now. I only ever have 1 pod per service and I deploy pretty infrequently.

For me its because how much easier Kubernetes makes my life. I'm using digital ocean's managed kubernetes offering. This means I don't have to worry about the cluster. The OS and updates and all that are managed for me. Same thing with cluster scaling. I just keep adding or removing services and they make sure the cluster is right sized. Basically I just dont ever have to worry about infra.

The next nice thing is the community support. Do I want a load balancer/reverse proxy with auto certificate renewal? It's a simple install command away from running. How about a HA database? MFA? Block storage? It's like magic.

The third thing is that most services are going to have a pretty similar deployment/testing strategy. So when I want to stand up a new one. I can copy 2 or 3 files from another repo, do a find and replace on the App name and port number and deploy a new service with just one command. Again, its like magic.

While I dont use this feature, I like the fact that I know I can move my setup to anywhere I want. Move my cluster into my house? on another cloud provider? This would be really easy to lift and shift.

> The OS and updates and all that are managed for me.

I can see this being a nice feature! I feel like the updates have never been a big issue for me in the past though unless it breaks some sort of dependency. In a typical Linux server `sudo apt-get update` has never been a big pain point for me :)

> I can copy 2 or 3 files from another repo, do a find and replace on the App name and port number and deploy a new service with just one command. Again, its like magic.

I just looked it up because I was curious, and it looks like setting up an Apache server on Linux is pretty much this same process, as in copy a couple files and tweak a couple settings. Also, I think my biggest question is if you're only using this for home services why can't they all live on one server and deal with it in a traditional style? Why does each service need its own environment as opposed to one environment with multiple services?

I feel like Kubernetes is hoisting all the traditional problems of deployment and infrastructure up to another level. Which is basically the definition of accidental complexity, adding additional complexity where none was warranted.

Once again, I can see how this is very useful if you're Google and need to ensure that if a service goes down it can spin itself back up on an arbitrary machine, but 99% of homebrewed services probably don't need this capability.

Then again, maybe I just enjoy the infra side of things and hacking together a build system and this is just my bias speaking :)

The cloud provider might handle OS and platform upgrades for you, but don't forget about updating your images! :)

Then again, dealing with that is also something Kubernetes can help with.

> I still don't understand why every application must be able to scale to infinity and maintain 99.999% reliability. Not everyone is Google and I feel like we'd be a lot better if if the not-Google companies just recognized that instead of layering on accidental complexity in the form of infinite scaling for your service that gets all of 5,000 requests a month.

I agree with this, but some are using Kubernetes (or other orchestrators) just to be able to utilize containers effectively across multiple nodes.

Containers absolutely solve many problems with application packaging, configuration, deployment, healthchecks, resource limits, port allocations, storage persistence, running multiple versions of the same software concurrently and managing more of the deployments as code.

I've actually written about my personal experiences and why containers helped me before, "My journey from ad hoc chaos to order (a tale of legacy code, services and containers)": https://blog.kronis.dev/articles/my-journey-from-ad-hoc-chao...

It's just that Kubernetes is the most popular solution for orchestration, which is why many end up getting pigeonholed into using it, especially if they need a managed container platform in the cloud. Finding hosted/managed Nomad or Swarm is an exercise in futility for the most part.

I agree very much about the accidental complexity of containers. Ignoring the runtime concerns (cgroups, namespaces, networking, etc.), the main problem seems to have been "we can't figure out how to get Python code to production", and the solution was "just ship a disk image of some developer's workstation to production". To do this, they created a new file format ("OCI image format", though not called that at the time), a new protocol for exchanging files between computers ("OCI distribution spec"), a new way to run executables on Linux ("OCI runtime spec"), and various proprietary versions of Make, with a caching mechanism that isn't aware of the actual dependencies that go into your final product. The result is a staggering level of complexity, all to work around the fact that nobody even bothered trying to add a build system to Python.

Like the author, I tend to write software in a language that produces a single statically linked binary, so I don't need any of this distribution stuff. I don't need layers, I don't need a Make-alike, I don't need a build cache, but I still have to go out of my way to wrap the generated binary and push it to a super special server. Imagine a world where we just skipped all of this, and your k8s manifest looks like:

    containers:
    - name: foo-app
      image:
      - architecture: amd64
        os: linux
        binary: https://github.com/whatever/download/foo-app/foo-app-1.0.0-linux-amd64
        checksum: sha256@1234567890abcdef
      - architecture: arm64
        os: linux
        binary: https://github.com/whatever/download/foo-app/foo-app-1.0.0-linux-arm64
        checksum: sha256@abdcef1234567890

Meanwhile, if you don't want to orchestrate your deployment and just want to run your app:

    wget https://github.com/whatever/download/foo-app/foo-app-1.0.0-linux-amd64
    chmod a+x foo-app-1.0.0-linux-amd64
    ./foo-app-1.0.0-linux-amd64

I dunno. I feel like, as an industry, we spent billions of dollars, founded brand new companies, created a new job title ("Devops", don't get me started), all so that we could avoid making Python output a single binary. I'm not sure that random chaos did the right thing, and you're right to be bitter about it.

You /can/ package python applications into a single binary -- just a zip file prepended with a python shebang. You can do much the same with the base python libraries. No one does it, I presume, because the community is against vendoring dependencies that way, the same way they are against static binaries.

https://bwackninja.github.io/2020/11/25/packaging-python.htm...

That doesn’t work for python extensions used by the application. These extensions may include or link to other dynamic libraries.

That doesn't solve anything. You still need a Python interpreter. It's literally just another way to move code around.

If you have a packaging problem approaching any kind of complexity all python packaging is trash.

This is effectively what Nomad permits - scheduling things that are not Docker containers.

> The fact that you need a special distribution like minikube, Kind, k3s, microk8s, or k0s to run on a single-node instance. (The fact that I named five such distributions is a canary of its own.)

They serve completely different purposes though.

Some are Docker-based designed to be lightweight for testing e.g Kind, others are designed to be scalable to full clusters e.g k3s.

And I think having the choice is a good thing as it proves that Kubernetes is vendor-agnostic.

> The fact that I named five such distributions is a canary of its own.

Seriously? Linux still can't even agree on tooling to display UI to the user, and its been like 30 years.

Or how many package managers and their supposed replacements are there?

> The fact that I named five such distributions is a canary of its own

That can be said for programming language implementations. Or programming languages as a concept, too. Or configuration formats. Or even arbitrary products or libraries that overlap and handle the same domain. Another comment mentioned Linux distributions.

My points is: what is your point?

The whole industry is like this (for better or worse). Picking something you don't like and slapping "it has more than one implementation" is hardly an argumentative against it (or any of the other implementations).

It's also not even accurate. You can absolutely run a single node cluster with kubeadm - aka upstream Kubernetes

> Through Brooks’ accidental-vs-essential lens, a lot of discussion around when to use Kubernetes boils down to the idea that essential complexity can become accidental complexity when you use the wrong tool for the job. The number of states my microwave can enter is essential complexity if I want to heat food, but accidental complexity if I just want a timer. With that in mind ...

I'm twisting my mind trying to grasp this interpretation in Brooks' complexity paradigm. I'm sure Brooks would be interested to learn there exists so called wrong-tools that can reduce essential complexity to accidental :). I think Brooks would put it that the essential complexity is the same and irreducible, but the accidental complexity is increased when the wrong tool is used.

> The number of states my microwave can enter is essential complexity

Two dials to set power and clock-timer is enough. Microwaves with digital panel and led display are a perfect example of accidental complexity.

Consumers don't stop and think about what they really need, or what the simplest solution to their problem is. They equate "more features" to "better." I think it was Joel Spolsky who said (paraphrasing) "nothing we did with our software ever created more new revenue than adding features."

Also see closely related paper:

http://curtclifton.net/papers/MoseleyMarks06a.pdf

This one inspired our most recent system architecture.

Thank you for sharing this, much more insightful than the article

A Kubernetes based architecture exposes to the developer and the operator a lot of things that superior hardware can abstract almost completely. Things like Mainframes can offer performance and uptime that easily beats pretty large commodity hardware clusters yet we seem to be pushing in this weird direction as an industry while ignoring why the mainframes came to be.

Think of Kubernetes as a "software defined mainframe".

It's an alternative implementation of many of the same concepts. You don't have to go to IBM and purchase a monolithic box with POWER processors in it.

But the thing is that mainframes abstract a lot of the fault tolerance in hardware thus absolving the developer of a lot of the responsibility and the cognitive overhead.

A mainframe in software pitch sounds a bit like that 90s RAM expansion software. Could it possibly work? To a limited degree, perhaps. Would I want it instead of an extra RAM stick? Hell no!

Not software emulated, but software defined.

For example, you can get a pair of enormous 100 Tbps core switch/routers for your data centre from Cisco. Upgrading these is terrifying, they cost an incredible amount of money, and if one fails you lose 50% capacity. Conversely, if you run them active-passive, you waste 50% of your capital investment.

Alternatively, you could use a cluster of 1000 VMs on commodity hardware to run your traffic management, with the rules defined purely in software. This can be an order of magnitude cheaper, scale to infinity, and allow smooth upgrades and change rollouts.

The former is what every large enterprise does, the latter is what the public cloud hosting providers do. You can't argue that it's some sort of "fake substitute" when the clouds operate production networks at a scale that far outstrips anything you'd find in any private data centre!

Similarly, many large enterprises use and prefer monolithic mainframes, but in the cloud the preferred option is something like Kubernetes.

For example, with K8s on a typical cloud, it's possible to use "spot" instances to temporarily scale out workloads at night for batch processing to use tens of thousands of CPU cores. No on-prem mainframe can compete with that on any metric. Not absolute throughput, not price, not price/performance.

I don't think AWS/Azure/GCP offer mainframes.

The more I think about it, the more I realize that generic declarative style, despite sounding very promising, might not be a good fit for modern deployment. Mainly due to technology fragmentation.

There's no one true way to deploy an abstract app, each tech stack is fairly unique. Two apps can look exactly the same from the desired state perspective, but have two extremely different deployment processes. They might even have exactly the same tech stack, but different reliability requirements.

Somehow you need to be able to accommodate for those differences in your declarative framework. So you'll pay the abstraction costs (complexity). But you will only reap the benefits if you're constantly switching the components of your architecture. And that typically doesn't happen: by the time you get to a point where you need a deployment framework, your architecture fairly rigid.

Maybe k8s makes a lot of sense if you're Google. But 99.99% of companies are not Google.

I'm not sure what you're trying to say. You list a number of problems that are addressed relatively well within Kubernetes whilst staying somewhat ergonomic; or at least more so than Chef, puppet, ansible, etc.

I'm not sure if you had the misfortune of working with any of the others, but it's extremely frustrating having to untangle them, especially after the 50th variation of it, because it's always done subtly different either because of scripts or something else. Kubernetes and Kustomize remove all these issues. You learn the language (or schema, I guess) once, and your never have to re-learn idiosyncrasies, dialects, variable nomenclatures, and others used between different projects.

Also, here's little room for creativity with templating languages that break tooling (I'm aware of Helm, but I also dislike Helm personally). I also have a feeling this is what people dislike about Kubernetes. It's boring, but it's meant to be.

> One way to think of Kubernetes is as a distributed framework for control loops. Broadly speaking, control loops create a declarative layer on top of an imperative system.

Finally: a post about Kubernetes that actually understands what it fundamentally is.

> Kubernetes abstracts away the decision of which computer a pod runs on, but reality has a way of poking through. For example, if you want multiple pods to access the same persistent storage volume, whether or not they are running on the same node suddenly becomes your concern again.

I'm not sure I agree that this itself constitutes a leaky abstraction. Kubernetes still abstracts away the decision of which computer a pod runs on, the persistent storage volume is just another declarative constraint on where a pod can be scheduled. It's no different from specifying you need "40" CPU cores, and thus only being able to be scheduled on nodes with >40 cores.

> There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code.

I agree - but you can do that with 'containers'. You just create an empty "FROM scratch" container then copy your binary in. The end result is that your OCI image is just a single .tar.gz file containing your binary alongside a JSON document specifying things like the entrypoint and any labels. Just like a shipping container, this plugs into anything that ships things shaped like that.

You get a bunch of nice stuff for free with this (tagged + content addressable remote storage with garbage collection! inter-image layer caching!), even if you're slinging about WASM binaries I'd still package them as OCI images.

> The use of YAML as the primary interface, which is notoriously full of foot-guns.

There's a lot more to say here, and it's more of a legitimate criticism of Kubernetes than most of the "hurr dur k8s complex" critisisms you commonly see. The ecosystem has kind of centered around Helm as a way of templating resources, but it's... horrible. It's all so horrible. A huge step up from ktml or other rubbish from the past, but go's template language isn't fun.

But I'm not sure how it could have gone any differently. JSON is as standard and simple as it gets and is simple to auto-generate, but it's not user friendly. So either the kubernetes getting started guide starts with "first go auto-generate your JSON using some tool we don't distribute", or you provide a more user-friendly way for people to onboard.

YAML is a good middle ground here between giving users the ability to auto-generate resources from other systems (i.e spitting out JSON and shovelling it into a k8s API) and something user-friendly for people to interact with and view using kubectl/k9s/whatever.

I'm confused, I thought JSON was created because XML wasn't user friendly. I on the other hand see no user friendliness in either YAML, JSON or XML. Just formatted text with or without tabs. Users don't like standards so whatever you choose somebody will complain.

I often wonder how many footguns could've been removed if projects like k8s/ansible used TOML instead of YAML.

Not many? I use k8s extensively and don’t see YAML vs TOML being anything that could make any sort of difference. It’s just a configuration file and it could be markdown or json or whatever and it wouldn’t make that much of a difference because it gets linted anyway. In the end you’re really just discussing the ups and downs of whatever markup language got chosen.

TOML is awful substitution in the ansible context. Let's play a game, how about I give you a real-life playbook and you translate it to TOML?

A person by the name of Venkat Subramaniam actually did a pretty nice presentation about accidental and domain (essential) complexity a few years ago: [video] https://www.youtube.com/watch?v=4MEKu2TcEHM

It's a bit humorous and long winded, but personally I rather enjoyed it and it struck a chord with me, especially the bits about how even things like configuration can be a source of frustration.

In regards to Kubernetes, of course, there's also lots of nuance and cases where it is good, as well as those where something simpler (like Nomad/Swarm/Compose) might be a better fit.

Either way, personally I think that containers are pretty good overall, unless you want to ship Vagrant images or full VM images.

Having worked in two of the largest cloud providers there's only one thing I can say which is that wherever I've seen kubernetees been used it has mostly served to make understanding of the application 10 time more complicated ensuring you spend 10 times longer to fix deployment issues

Once you finally get it working is "ok" at least until you have to reconfigure it...

Not like you couldn't have launch a few containers on "restart always" mode... and no, this is not me not understanding autoscaling and the like, IMHO I think that goes to the realm of accidental complexity since the use cases for that are more a corner case than a necesity for 95% of the scenarios where I've seen it being used

> When you run a container on Kubernetes, you don’t specify which computer they run on; you (hopefully) don’t care.

And then you want to deploy something rack aware and you have to deal with `requiredDuringSchedulingIgnoredDuringExecution` in your yaml. Even greater fun ensues if you need some kind of special-purpose hardware.

Also if you have both Windows 2019 and 2022.

Or if you have licensing based on "hardware cores" or "servers".

Kubernetes and its complexity again..

Reply on news.ycombinator.com