Containers are not just for Kubernetes
ramansharma.substack.com
ramansharma.substack.com
Is Kubernetes really that popular out there in the wider software world? Or is this author just in a bubble?
If so: Kubernetes is kind of like a fancier version of that which can scale to a large cluster, along with tools for handling common tasks associated with these deployments like name resolution, HTTP/S ingress, automatic scaling, and periodic tasks. It scratches a lot of itches for large-scale production environments.
I guess it could be a regional thing, but for the most part i'd be inclined to say yes.
Most of the larger cloud vendors offer managed Kubernetes in one way or another (Azure, GCP, AWS, Scaleway etc.), which you cannot say about any other orchestrator out there.
Most of the large enteprises that i've worked with also are moving towards introducing Kubernetes internally, sometimes with pretty large clusters for multiple projects at once!
Similarly, most of the tooling out there is focused on Kubernetes nowadays (networking, storage, deployment solutions), which isn't the case for other orchestrators.
Of course, many out there don't even need an orchestrator - Docker or even Docker Compose (or the podman equivalents, or other solutions like that) are oftentimes enough for smaller scale projects, it can be a while before you would enjoy the benefits from the likes of Docker Swarm, Hashicorp Nomad or even one of the many Kubernetes distros out there. Once you need to scale past one (or a few servers) and need to orchestrate deployments across all of them, it might just be worth to look into them, though.
I guess that's because Google largely won the "container wars", being the first to market with a reasonably complete feature set, brand reputation and resources for continued development, offerings by most cloud vendors and also a large dose of hype in the form of developer talks, demos, press releases/newsletters, tutorials, learning resources and i guess even certifications. Of course, that wasn't all just Google - once the hype around it started, numerous parties joined in, wanting recognition, relevant skills, or even to market their products that are built around supporting the complexity of Kubernetes.
Compare that to something like Apache Mesos, which, while initially promising, largely fell by the wayside and never really caught on: https://mesos.apache.org/
Larger surface area that can be controlled programmatically without needing to involve the people managing the hardware or network. This makes certain engineering choices (like microservices) more tenable.
If you’re big enough to have a Unix team and SRE’s then today I’d wager kube can save you time and money vs. manually managing cpu, memory & disk allocations, software deployments/rollbacks and disaster recovery steps.
If you’re smaller than that, it really depends on the skillset of the team as to whether kube is a good idea or not.
https://ramansharma.substack.com/p/the-case-for-cloud-server...
Amazon ECS launched April 2015 and EKS June 2018.
I remember the docker deployment story being a mess. You still needed to manage a fleet of containers the same way you did VM instances.
I work on the Kubernetes offering for one of the major cloud providers... yes, yes it is.
https://cloud.google.com/sdk/gcloud/reference/compute/instan...
(I'm a googler, but do not work anywhere near your cloud. I used this for a personal project.)
The process of switching to using simple vms was a gcloud one liner to spin up a vm with the same docker image. That basically worked the first time we tried it. We ran with this for a few months. Then we finally sat down and got ourselves a load balancer with an instance group and an instance template. We deploy by telling gcloud to replace the template on the instance group. It does a rolling restart and it uses our health checks to ensure it does the right things. And it's stupidly easy to script together. We use Github actions for automated deploys. It does rolling restarts, has health checks, monitoring, logging, etc. built in. Everything you need to run a simple monolith.
It's similar to what Amazon did with ECS. Just less convoluted and way more easier to use.
People assume they need all this crazy complicated infrastructure. But it's all commodity off the shelf stuff these days and it doesn't have to be that hard to set up. Could we automate the above? Sure. But what's the point. It's 1 command to create instance templates, another command to tell the LB to run it. Provisioning the LB with a wild card certificate is a five minute job using the UI. It will tell you the commands if you need them. But it's something you do once per environment.
https://ramansharma.substack.com/p/multiple-stops-on-the-clo...
[0]: https://cloud.google.com/compute/docs/containers/deploying-c...
Most of the problems they claim to solve where already solved years ago by more mature solutions:
- reproducible builds and dependencies management: distribution package managers
- high availability: load balancers, keepalive, dynamic routing, clustering and most importantly: data replication (where is this in k8s?) since downtime is nothing compared to data loss
- resource limitation: cgroups
- isolation: the good old user and groups, vlans, firewalls
- orchestration, rollouts, rollbacks: CD pipelines, configuration management
Technically the only thing I find easier with k8s is bin packing.
The real advantage I find in containers and k8s is they became a standard. You don't have to learn the architecture that each company has designed for itself and dig into custom built solutions. So you can easily switch jobs and they can easily replace you. If you throw that away with custom container implementations for each PaaS offering and fragmentation of orchestrator solutions then you only get the complexity and no benefit. We're going to see in a few more years where it goes but I'm not exactly optimistic.
Source: I fix people's infra when it's fucked. By far the most fucked to date, are Kubernetes/etc put in place to solve problems solved with much simpler solutions.
Something is not right if at the start of a planned outage, it takes your staff who "built" the system in place an amount of time measured in hours to:
- block most public traffic; while
- allowing traffic from a restricted set of IPs; and
- showing a 'nice' maintenance page to the blocked visitors
I don't honestly recall the exact time it took them to make the above changes, but it was somewhere in the order of 3 hours.
The whole point of "DevOps" is to promote things like continuous delivery. Smooth, automated deployments. You can't tell me that "we need to deploy a change, that requires a migration be applied to the stored data, and must prevent user access while this occurs" is not somewhat common.
That's not to avoid the work of creating maintenance pages of course, it's to avoid needing them.
And it sounds like it works for you, but I don’t think I’d choose that route, and I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations.
For sure. I'm just making the point that to do continuous delivery you wouldn't necessarily need maintenance mode pages, which predate continuous delivery.
If your development team is sufficiently experienced to do allow for the no-downtime approach that’s a big win, but my experience so far makes me pessimistic about how widespread that kind of experience is.
Edit: I should clarify somewhat: I’ve had to introduce at least one of the following concepts at basically every client: continuous/automatic deployments; automatic migrations; automatic maintenance pages during outages (planned or otherwise); and hard as it is to believe: version control.
My clients are typically smallish, so anything I can do to solve problems/provide solutions at the infra layer is a no-brainer because setting up an automated {enable maintenance mode; apply migrations; disable maintenance mode} is a lot more reliable long term than trying to teach every possible developer how to handle backwards compatible models - particularly when the clients application language of choice is something I’m not necessarily experienced in.
No, I have a job.
Seriously though, no. As a one-man band, anything besides "client X needs Y" or "[internal/forked/contributed to] tool A needs enhancement/bugfix B" tends to be very low priority.
While that's a fair position to take, you also might be in the minority here. I've seen plenty of in-house solutions in corporate environments and one of the things that they've all had in common was how bad they were, due to them being a way to just get things done and no more (e.g. no proper testing, no proper documentation, bad/wrong abstractions).
It's not like your (or anyone else's) collection of Bash scripts can compete with open source solutions that have had thousands of collective man years put into making them better and addressing the never ending churn of keeping things working across new distro releases etc.
I'm not advocating to go straight to Kubernetes - Docker Swarm or Hashicorp Nomad, or even just Docker Compose can all be decent alternatives that are simpler to wrap one's head around and manage, in most smaller deployments.
As for no one knowing what containers do, i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat, with who knows what sort of Cthulhu living in the shared libraries folder and everything being so brittle that you cannot even look at it the wrong way, or it'll break and you'll spend days digging through obscure error logs. Worse yet, have you ever needed to manage an environment that has applications that need 4 different JDK versions and many different launch methods (some use systemd services with Tomcat, some use Tomcat launch scripts, some have just a .jar that needs to be launched etc.), yet still count upon some shared environment configuration being present? Give me individual containers over that mess any day.
I think that there should be a clear separation between what a server needs to keep operating and where the "business software" would run - basically an /app directory with a subdirectory per application with whatever it needs, be it bind mounted data storage, configuration files, log output (in lieu of log shipping) etc., as opposed to having some stuff be strewn across /var, /etc, /opt, /www and who knows how many other directories. Containers give me just that.
I'd be astounded if I'm not. The tech industry is ridiculously cargo-culty. I never said I expect the majority of people to agree with me, I said what I prefer.
The majority of people probably think McDonalds is just fine, and that Google is "not so bad" too, and it doesn't bother me that other people disagree with my opinion.
If you ask me to fix your fucked infra because someone threw together a Rube Goldberg like Kubernetes environment just to achieve rolling deployments of their application, I'm going to tell you it's a shit solution, just as if you asked me to critique a McDonalds hamburger I'm going to tell you it's a shit hamburger. Me saying it's shit doesn't mean you can't use the thing, it means my opinion is that there are better ways to solve the problem/achieve the desired outcome, and you can either accept that opinion and make a change, or disagree and I'll go provide my services to another client. It's really that simple.
> thousands of collective man years put into making them better
They've had a ridiculous large number of man years working on them, no question.
The whole point is that we disagree about whether they're "better" or not. Adding complexity is not necessarily making something better.
> i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat
In... a decade and a half of doing this I think I've been asked to support one JVM based thing, and that was quite a while ago. I'm sure it is a shit show, but I'm also sure that there are better alternatives than 8 more layers of abstraction.
Lightweight VM-like containers are literally a thing that exists, and can give essentially the same provisioning as a traditional VM, with the "lightweight" / "shared resources" aspect of a container solution.
Although you will get with any solution at a point where text configuration is too large and you get into templating and then coding, and you will get there with K8s as well - see pulumi, dhall, etc
There's people who can handle complexity with an elegant solutions and people who can't and search for silver bullets like K8s.
As in? What distro package manager is allowing me to give a project newbie a single command to hit the ground running across all platforms, "docker-compose up"? What distro package manager is allowing me to have several different platforms that can just grab a GitHub repo link and deploy it viola?
The different platforms for deploying is about cloud-agnosticism, avoiding vendor lock-in by using an easily reproducible standard. Even for just one cloud deployment, Ansible and similar tools aren't as simple and reproducible as a docker-compose.yml and Dockerfile.
I'm not saying I've started from 0 and tried all the comparable solutions here, comparing and contrasting them. It's that when I looked at the lay of the land, asked "How do I get a Django app easily reproducibly built and deployed, across dev and prod, the Docker solution that I went with appeared to be such a shortcut to the rest.
As a devops I strongly oppose developing on the desktop OS. Only your repos and code editor should be on your workstation, and you should have dedicated dev environment in a server farm - be that vms or containers.
> "How do I get a Django app easily reproducibly built and deployed, across dev and prod, the Docker solution that I went with appeared to be such a shortcut to the rest.
Only you don't deploy Docker in prod, do you? The fact that the container starts means nothing if it doesn't interface with the other contanerized apps. So you start needing a Kubernetes cluster locally and all goes to hell from there.
Vagrant is a really good solution for this: it is generally VM based, but the same "box" (basically an image) can be build for multiple providers, so you can use Docker or LXC as your provider, I can use Parallels on a Mac, Jenny can use VMWare Desktop, Billy can use HyperV, and the casual tester can use VirtualBox.
This also doesn't preclude the use of covering different architectures for the boxes: years ago I setup the tooling for a project where one developer had a laptop without a 64bit-capable CPU, so most of us ran an amd64 box, but his was i386.
Nix (it's the "distro package manager" of NixOS, but runs happily on any Linux distro, without even needing root permissions; it's also supported on macOS. I've seen references to other Unix systems like BSD, for WSL, and possibly other OSes, but they aren't officially supported).
In fact, we don't even use "a single command"; our projects have .envrc files for direnv (which has Nix support out-of-the-box), so our shells automatically enter/exit a (reproducible) development environment just based on their current working directory.
> "docker-compose up"
I wasn't aware that Docker was cross platform? I thought it used Linux containers, which requires running a VM on macOS, etc. (e.g. that's what applications like "Docker Desktop" do in the background). Apparently it also supports "Windows containers" now, but those aren't even compatible with the Linux ones!
> What distro package manager is allowing me to have several different platforms that can just grab a GitHub repo link and deploy it viola?
Nix does this with a built-in function called `fetchGit` (there are also specialised functions like `fetchFromGitHub` and `fetchFromGitLab` which download the tar.gz of a given commit, rather than checking-out the repo).
"Deployment" varies, depending on what you want; e.g. it could be a simple 'nix-copy-closure' (to a physical machine, a running VM, etc.), it could be making a container image (with a call to `ociTools.buildImage`, `dockerTools.buildImage`, etc.), or a VM image (VirtualBox, libvirt or Xen). NixOps and Disnix can orchestrate and deploy those, but I've not tried them (we use AWS ECR and CloudFormation).
[1] https://github.com/NixOS/nixpkgs/issues/95903
[2] https://github.com/nix-community/home-manager/issues/2266
From skimming the comments, it looks like Nix itself works; although there seem to be many packages in the Nixpkgs repo which don't have that architecture/OS in their metadata.
Never bothered with Docker or Kubernetes, as I already have my needs solved.
Call me a dinosaur if you like, but it’s fundamentally simple and it works for me.
To expand on the title: Containers are not just Docker.
I feel like Kubernetes tried doing way more than that: it added additional abstractions like ingresses and load balancers, abstracted away storage with its drivers and PVC concept, added support for all sorts of cases when something should be run (e.g. if there's a task you'd like to schedule or even just run once), introduced namespaces and its API is pretty extensive and extensible, especially with CRDs. Just look at how many projects have propped up around it, from networking solutions (Calico, Flannel, Istio, linkerd, ...), to storage solutions (Longhorn, Rook, ...), to all sorts of additional software, like Helm, Kustomize, Skaffold etc.
Of course, that's not to say that everyone needs all of that complexity, as others have rightfully pointed out in the comments. Personally, i think that the sweet spot is focusing on OCI as the standard to go for and hosting most of the software that you need inside of containers with a lightweight orchestrator, only moving to something more complicated once the need actually arises.
For example, Nginx in a container on Docker Swarm is far more simpler while also being a bit more flexible than Kubernetes ingress: just this week i needed to expose a Sonatype Nexus Docker registry on server port 5000, 5001, ... whilst also wanting the reverse proxy/ingress to handle SSL/TLS termination with a wildcard cert. Kubernetes ingress doesn't support these non-standard ports, whereas NodePort would have necessitated running on the 32000-... port range and even then i'd need to setup a separate Nginx instance if i wanted to use it for certificate management, as opposed to importing the certs into Nexus itself, which kind of defeats the purpose. Also, needing to trick a Traefik ingress (the default in k3s) into using a custom default wildcard cert necessitated a HelmChartConfig, ConfigMap, Secret (TLS) and TLSStore, which seems badly documented, cryptic and also just wasn't too user friendly - and felt like a step back when compared to just needing a nginx.conf (or the Traefik equivalent) and the actual .crt and .key files.
Whereas with Docker Swarm, i can just have one Nginx container and expose whatever ports i want, because i'm not pigeonholed into a particular type of abstraction - however, that comes at the expense of not having easy to configure ingress, since i need to change the Nginx config manually. Regardless, the simplicity of Docker Swarm and the likes of Hashicorp Nomad is actually a boon - in the case of the former, pair it up with Portainer and it should be pretty smooth sailing from there onwards, even easier than using Rancher with Kubernetes!
Admittedly, it all probably depends on the scale that you operate at - because once you really need to think about node draining, resource rebalancing upon adding new nodes, autoscaling and other advanced concerns, then you might want to look into Kubernetes, but for the most companies and projects out there, that's hardly necessary. For most of them, regular containers with some lightweight orchestrator running on top should be more than enough. I think Docker here wins out as the runtime as well as opposed to containerd or podman, because of its historical significance and how much software has been build around it (hell, even Rancher in Docker basically takes the Kubernetes in Docker approach which is actually really nice, apart from it all breaking on newer RPM distros because of a lack of cgroups v2 support).
Now, there are other container technologies out there as well (e.g. LXC or some might even suggest the BSD jails), but i think that we're not in too bad of a spot right now. Luckily alternatives to Kubernetes are still maintained and are pretty usable, sometimes even Docker Compose can be enough for prototyping or dev environments and if you ever find yourself needing to look into using Kubernetes, projects like k0s or k3s are pretty much excellent for that. Of course, i'd also suggest a modicum of supporting software, like Rancher for a nice web UI and Rancher Desktop to not have to pay for Docker Desktop, if you don't need it.
In my eyes, this approach also fully eliminates vendor lock - all i need is a bunch of VPSes on any cloud host, on which i can initialize a container cluster with the likes of Ansible and migrating is similarly easy (especially with bind mounts for storage). PaaS platforms can be nice for when you don't want to deal with managing your own servers, but i much prefer to keep my apps as portable as possible. That's not to say that the PaaS approach is not valid, as it can indeed save a lot of time and effort, if you haven't invested in automating the setup and management of your own nodes!