Solving Common Problems with Kubernetes

What’s ironic, too, is the complexity is so far abstract from the problem it’s solving (and the problems it then introduces) that troubleshooting becomes a nightmare.

In my experience, too; few pushing adoption/application of these technologies are system administrators, or even architects/designers - or are only so fresh out of uni (which i do not mean to suggest is bad..)

The accountability becomes really muddy, not even considering security boundaries and other multi-business-level disciplines these “solutions” impact.

I get it. There’s a place for these technologies. But almost every adoption I’ve experienced to date is more of a shiny-thing syndrome.

So; thanks for this comment.

This is such a great description (I also for my job work exclusively on a platform based on Kubernetes). I ran K8s at home successfully for a while using K3s (such a great project/tool) to become more proficient. After a few months I found that there were so many features I didn't need for a homelab that the complexity wasn't worth it.

I feel like, at the moment, Kubernetes is awesome when you have two ingredients:

1. You go with a managed K8s offering (such as AWS EKS)

2. You have a team of engineers dedicated to the health of the K8s platform

It's pretty cool that a somewhat small platform team can wield the scaleability of a company like Google, BUT they need to be very good with K8s. And many workloads probably don't need that much scaleability.

For some miraculous reason K8S is ubiquitous and everybody uses it. Standardization is a boon.

People complain about git using similar arguments, and yet having a vast majority of the tech world on it is a boon for tooling.

Both those technologies are excellent but take some time to master (you don't have to be an expert though).

I can get on a project using git, kubernetes and say RoR and understand it very quickly without help. It is well bounded. It takes a git account and a kubeconfig. All set.

A custom python codebase deployed on custom servers running in big enterprise-y network, not so much.

Isn't that what a container is though? (static linking here meaning everything inside the container and no need to provide & manage external dependencies).

The container format has the advantage of simply being a standard linux environment, the kind of thing you work with anyway and can host any process type or application.

As for k8s, of those 100s or 1000s of lines of yaml, there's probably only 20 or so that you're interested in for a basic deployment. Tools like kustomize or pulumi make templating simple.

K8s is pretty simple under the hood. As an exercise, I once set up a cluster by hand on a couple of vms using kubeadm. It all dropped into place for me, understanding what kubelet was doing. K8s seemed a lot more sane for me thereafter.

K8s' fatal flaw was being too big for its britches. As individual pieces, none of the K8s services are useful. Nobody downloads and installs just one of the components for use in some other project. And for that reason, it will always be just one big fat ugly monolith of fake microservices.

The architecture that will come afterward will be the model we've had since the 90s: single components that do their job well, that can be combined with other components in a hobbled-together way, that together make up a robust system. The same thing that lets you build an architecture out of PostgreSQL, Ory, Nginx, Flask, React, Prometheus, ELK, etc: they're all separate, but they can all be combined the way you want, when you want. And they're all optional.

If one looks from operations/sysadmins perspective: - I have a big server or 10 servers - I don't want to install PHP/Python/.NET for each dev team on the server or get new server and maintain it separately for PHP teams and separate way for .NET teams - I want to provide environment where devs can just drop a container so it enables them to have what they need without me checking off each install and each VM

I also don't think dev teams should run their own k8s clusters. If you have huge company and people running data centers you don't want them to deal with whatever flavor of back-end code these servers are running.

That's pretty much what happens in the Go space, and people are still deploying k8s like crazy there, seemingly failing to consider the trade-offs involved. It's people's full time jobs to manage all of this when it really isn't needed if your ops story was just a bit simpler.

I'm a developer, so for the most part I don't really care what the ops people are doing, whatever works well for them, right? But the way k8s works and because of the "dev-ops" stuff it's also pushed hard on developers. I've worked at companies where it was literally impossible to set up a local dev instance of our app and even running tests could be a challenge, because unless you're very careful k8s gets in to everything.

We need like what pytorch was to tensorflow, its clear that google has completely lost the plot again with this insane unmanageable complexity that blows up in your face every 5 minutes.

If you try to do something like infrastructure as code across multiple environments, with containers orchestration, service discovery, autoscaling, easy troubleshooting, authorization, easy domain name resolution, secrets at the container level then trying to do those things with App Engine becomes a horrifying ordeal.

The next question is obviously do you even need those things? But basic k8s really is easy to setup. Spinning up a cluster, writing some deployments, services, and setting up an ingress takes < 6 hours (half that if you do this regularly that you have documentation on it) and is the same experience everytime - there is no, I am using GCP at home so I don't know how to do this with AWS here. The CI/CD for the git repo is also pretty much the same.

Moreover, it's actually good. Sometimes you have industry standards that are just painful to use but do anyways because it's popular. With k8s, it really is a well thought out, mature solution that's better than most alternatives. Managed services often have a whole layer of bullshit e.g. last night I was looking at using App engine for a node app and I can't use the standard environment because I am using the latest version of Node that's not supported yet - and it's just one more nuisance to deal with. With k8s, it's just compute resources, containers, done. Simple.

If your lead engineer gets hit by a bus tomorrow the whole deployment is in a helm chart somewhere and you can hire generic kubernetes tech #158326 to take over that job at any time.

Being able to engineer the app correctly the first time around, plus guaranteed business continuity are pretty big selling points. I would not invest in any company that did not have kubernetes, or a similar orchestration platform underpinning their product. Eventually that engineer will leave and you do not want to be scrambling for their replacement in the middle of the big growth push.

On the other hand, once you know k8s, it means you don't have to learn the specifics of any cloud platform. For most of my career I've worked in places that can't use cloud providers. I honestly dread having to learn all the different GCP or AWS tools/apis at a future job. But now that I have spent a lot of time learning k8s, it's nice to know that I can build on any cloud platform in the future, or self-hosted servers, and reuse the standard k8s APIs that I use at this job.

If I was building a startup, I'd use whatever the majority of engineers were already familiar with -- k8s, aws, gcp, whatever.

K8s is also available as a managed solution with GKE Autopilot, which in general is a breeze to use.

> You never worry about the problems listed in this article.

You still have to. If you look at the whole list of "how-to" guides one of the language docs for GAE (e.g. [0]), you see that there are about as much topics to worry about than if you were deploying to K8s. They also pretty much 1:1 map to things in K8s, but in contrast are non-standardized across vendors.

> deploying it with minimal config

I think the article laid out quite well how for a minimal deployment, the configuration you need for K8s is also pretty minimal. Yes, it might be more than a one-liner for a GAE config, but most of it is boilerplate you can ignore.

[0]: https://cloud.google.com/appengine/docs/legacy/standard/go11...

* you’re not using GCP

* you have workloads that don’t mesh well with common cloud PaaS patterns

* you want to pick-and-choose your cloud dependencies

* you want to use a commodity tech-stack so that hiring, tooling and support are far more readily available.

All of this comes with the obvious caveat of: sure, if you aren’t having issues, or reaching the limits of whatever PaaS solution you’re currently using, then don’t swap unnecessarily, but that still means there’s a lot of useful features that K8s offers.

One of the upsides of k8s is that it gives you a general, platform-agnostic way to describe this, on AWS or GCP or self-hosted or whatever. The downside is that it's more flexible and therefore more abstract than any particular provider. The upside is that you only have to learn one set of concepts (in theory).

AWS ECS is basically a docker swarm cluster customised by AWS to include auto-scaling of service containers/nodes and generally integrating better with other AWS services.

k8s is that orchestration layer, but on anabolic steroids. Plus a standardised interface on any cloud platform you might need to run on.

With bigger muscles comes more complexity, more maintenance work etc. (was designed for orchestrating millions of containers, so way more complexity)

which is why everyone always moans about it. (including me)

It’s a very cool and very useful piece of tech when yielded correctly. Just don’t expect it to work like a “kitchen sink”.

Each of those have their own k8s implementation.

It's a standard way to define and run container workloads (jobs, services, etc) and related resources like storage, configs, secrets, etc.

(don't @ me just to say it's more complicated than that)

The team whose main objective is driving customer value only truly cares about the Ingress resource such as defining their routes to expose their work to the customer.

Unless you want every team to take the burden of logging, tracing, auth, etc you centralize around a global edge network which is where one would define a ingress controller resource.