Kubernetes for Developers Who Know How to Develop
blog.ali.dev
blog.ali.dev
It's a very cool system. I completely understand why people half-jokingly call it a "distributed operating system." It does a lot of things related to the lifecycles of various state (secrets, storage, config, deployments, etc).
However, I believe it goes way too far into putting infrastructure into non-cloud-managed state machines. Things that exist in most modern clouds are being reinvented in K8s. What's more, is that K8s objects are being created as interfaces to the underlying cloud objects. So you now have 2 layers of abstractions, each with their own quirks. It's too much.
Not to mention that IaC for K8s is extremely immature. This will improve, yes, for some definition of "improve." But if you've ever written Helm charts that integrate with Terraform, you'll know about all the spinning plates you have to keep balanced.
It's not a system I see sustaining into the long term future. Google may continue to use and support it forever, but afaik, they are the most invested in its success. Other cloud platforms, like AWS, seem to be focusing on not re-inventing all of their cloud offerings in K8s.
My more optimistic view is that vendor specific APIs are being standardized. Initial implementations have their issues, but as more people use them the cloud vendors will improve their offering.
The market wants cloud computing to be a commodity. I expect it will get what it wants via k8s or some other technology.
At the very least we can actually simulate our stack locally.
K8s is hugely supported and invested in by Alibaba, and TONS of companies you've never heard of.
All of these companies have huge, mission critical reasons not to use AWS/Azure/GCP and invest in K8s.
From an end user's standpoint, I am certainly not going to dispute your criticisms about Helm.
Google's original design of K8s seems like it was originally designed to be generically worse than GCP.
However, from an ecosystem standpoint, K8s is a project I'm very confident will be around for a long time.
Very true. The problems occur when smaller companies without these mission critical reasons make the same choices because the others are making them.
But yes, "roll your own K8s" is for larger organizations.
It turns a tool that escaped the tyranny of endlessly mutating blessed servers with immutable, contained services and unified declarative life cycles, back into an imperative mess of magical incantations that must be spoken in just the right way.
Kubernetes is simple when used as designed, but staggeringly complicated when forced into the mutable imperative workflows it was expressly designed to prevent.
If you are using Helm to generate source code for your application you still have the added complexity of additional build step, but at least you can choose to add the generated code to your app in a way that tracks with the rest of your code.
Also most Helm chart authors are of varying skill level, and even skilled ones necessarily make incorrect assumptions about your deployment environment. It takes a lot of addition code in helm charts to support more flexibility, so it often get ignored, and you are left with a black box that doesn't quite do what you'd want it to do.
IaC?
I should fucking hope so. I don't want to change my helm charts because I'm deploying on Azure for this particular client.
> Whenever you hear Kubernetes, you’re probably going to hear Docker in the same sentence. Kubernetes and Docker are like peanut butter and jelly—they’re a perfect pair.
For what it's worth, this article was originally written in October 2021, but even at that point it was clear that Kubernetes would remove support for docker. In fact it has already with Kubernetes 1.24. Not exactly "Best Friends Forever". I wish people writing this kind of introductory blog posts would take the time and just quickly explain container runtimes instead of just mentioning docker and be done with it.
[1] https://cloud.google.com/kubernetes-engine/docs/deprecations...
The question was did they remove support for docker. They have not.
They simply removed dockershim and no longer special case it.
Your point is that we should stop using a brand name to describe OCI open standard containers. However, like aspirin, the names are interchangable for the vast majority of people.
OCI container doesn't really roll off the toung, and container by itself could mean all sorts of things.
Docker is really the best word to describe what people are thinking about it every day conversation. So much so that I think they are at risk of loose their trademark.
Have they not? Note that the GP asked for GKE specifically. The support page I linked to literally says so:
> GKE will stop supporting node images that use Docker as the runtime in GKE version 1.24 and later
Removing dockershim removed the existing support for docker, because docker does not support CRI (Container Runtime Interface), the API required by Kubernetes. You can go through a third-party solution that adds CRI support on top of docker, but most managed Kubernetes offerings simply removed docker support.
I don't see any argument supporting the claim that docker is the "best word" to describe containers. I am also not aware of ambiguity for the term "(Linux) container" when it comes to operating/deploying software. What else does it mean in that context?
While it's true that 1.24 does not support docker as the specific container runtime that's directly used by Kubernetes itself, this has approximately zero impact on how the vast majority of beginners would use Kubernetes, as out of the box you're still able to run docker containers.
Probably not the kind of confusing detail that needs to be in an intro to Kubernetes article.
No, you're able to run containers from images produced by docker provided it exports them in OCI format. At no point does k8s see anything to do with docker. Saying docker when you mean container or container image is incredibly misleading at best, and less charitably, is just plainly wrong.
Edit: Actually, it looks like if you want to add docker it's easy - https://kubernetes.io/docs/setup/production-environment/cont... - but you would have to install that support since it is not included out of the box.
Docker produces OCI images, there's no need to "export them" in that format.
So since Kubernetes can run any OCI image, and Docker images are OCI, Kubernetes supports running Docker images out of the box.
The documentation you linked to is if you wanted to swap out the container runtime Kubernetes is using, not if you just want to run a Docker image.
Fun though personal attacks are, you would be wrong; I have done all of this, including using docker to build images and k8s to run them.
> Docker produces OCI images, there's no need to "export them" in that format.
Sure.
> So since Kubernetes can run any OCI image, and Docker images are OCI, Kubernetes supports running Docker images out of the box.
...Kubernetes supports running OCI images out of the box. That they were built by docker does not make them docker images, any more than building a Windows program with MinGW creates a "Linux program" just because it was compiled on Linux. If you use docker to build an OCI image and then create a container from that image in a stock k8s cluster, you are not creating a docker container, you are creating a (most likely) containerd container from an OCI image.
> The documentation you linked to is if you wanted to swap out the container runtime Kubernetes is using, not if you just want to run a Docker image.
Yes, I was attempting to charitably include the case where your claim could still be correct. (Since by adding the docker runtime you can create a k8s cluster that creates docker containers.)
For most users, using docker through a CRI compatibility layer is not an option as they use some sort of Managed Kubernetes, and I am curious to hear which of those keeps supporting docker as container runtime.
https://kubernetes.io/docs/setup/production-environment/cont...
The dockershim removal FAQ says how to continue to use docker engine.
https://kubernetes.io/blog/2022/02/17/dockershim-faq/
-- Linux container could be LXC, systemd-nspawn, snap, flatpak, nixos-container, and many many many other things.
That's because Linux containers are built on Linux interfaces, but Linux itself does not have any prescriptive requirements on how to stich them together.
[1]: https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-...
There are tons of ways to build containers without Docker. You're effectively encouraging propagating confusion.
So I can't use Dockerfiles and `docker build` to build things for k8s?
Kubernetes used docker engine under the hood in the past. Now they abstracted useful part of this engine into API. There's implementation from the docker (containerd), there's implementation from Redhat (CRI-O), may be others. Docker don't have to be installed for Kubernetes to work anymore.
Building container images is a different topic. Kubernetes does not have anything to offer here. So you probably still need docker in your development machine and in your CI pipeline to build those images. There are plenty of alternatives rised in recent years, most prominent ones are kaniko, buildah/podman, but they're far from docker in their maturity.
That actually makes a problem. It's hard to run docker and kubernetes side-by-side. Or docker inside kubernetes. So if you want to run your CI jobs inside Kubernetes, there's no good solutions right now.
I think people will eventually migrate to Kaniko. It's from Google, it seems to be a sane approach. But right now it's a mess.
K8s abstracted out containerization in preparation for the anticipated long term migration of the ecosystem to Podman et al (lots and lots of people are invested in moving "containers" away from "Docker").
With that said I still prefer docker for local development by creating real ephemeral clusters with kind
honestly, your suggestion for container runtimes is a great idea - it's now on my list of blog posts to write! I'll be sure to credit you :)
I think the constant thought in the back of my mind is… Docker containers and Nginx is definitely enough for my needs. At work, it’s a different story, but more most things, it can be over kill.
So something that could be run with $5 droplet, suddenly requires $200.
They you want to install grafana, ELK, service mesh and whatnot.
You won't get a private LAN (so you'll need some firewall rules on the hoster side) and you'll have to do some setup to get everything served up on the right public IP address(es), but you don't need a dedicated load balancer product. Unless you expect your virtual servers to go down randomly, I suppose.
I've set up k0s on Oracle's free service to mess with. It doesn't need nearly as much as 12GiB of RAM. Give it 500MiB per machine to be happy and it _should_ run fine with 10.5GiB to spare unless you're adding tons of overlays and additional services on top of your applications.
As far as I can tell, the biggest cost with Kubernetes is maintaining the system and keeping up to date with the advancements and deprecations in the Kubernetes field. You can't just upgrade the Kubernetes version and expect everything to work, and every new concept that replaces anything old comes with Kubernetes layers of complexity.
I honestly don't know what so many companies are using all those resources Kubernetes provides for. I have a feeling people are so caught up in the "modern server management" world that they've forgotten how powerful and reliable a $25 VPS is these days.
[1] https://registry.terraform.io/modules/kube-hetzner/kube-hetz...
I have a nice server that I used for 10-20 applications like Time Machine backup, media streaming, file syncing, home automation, etc. I was looking into k8s, but it seemed way too complicated when I only have one physical server, so I went with Docker + Docker Compose instead.
All in all, I still think docker, docker compose and nginx is the way to go for hobbyist stuff. I doubt my ISP would let me host anything at a huge scale off my home router anyways.
If you start having more servers, a k8s cluster can do something like automatically transfer workloads when one machine has an issue for example, or dynamically allocating storage (even cloud) when needed, or allowing you to issue a single command to spin up more instance of whatever task you'd want to run more in parallel, or updating a version in a config file and letting the cluster do the job of stopping/updating/restarting things.
But what you also have to do is ensure the underlying networking between machines works, all versions of the binaries on all the machines of the cluster are in close agreement with each others, encryption of data via certificates and CAs is satisfactory, and your master(s) might require different maintenance from the workers - but you'll end up reading the docs to know how to roll updates step by step.
In your use case I would stick with a smart use of docker-composes, perhaps just centralized in a private repo, with cron jobs on top. If you buy/rent a couple machines of any kind (from raspberry pis to cloud instances) to put on your network and run things for you, then a cluster becomes increasingly interesting as the amount of tasks needed/wanted and amount of computing available grows.
My context: I have some limited experience with work (including some scary "abstractions" on top of K8s & Helm) and a recent CKA (certified kubernetes administrator hands-on certification) from LinuxFoundation.
To OP's point, this seems to be a trend among corporate blogs. Targeting devs with irrelevant content. A few weeks back I remember seeing a "SOC 2 for devs" blog article and was really scratching my head. That's not the level that devs work at, at all. If you're paying six figure salaries to devs to sit around all day worrying about overweight bureaucratic nonsense, or other tasks way beyond their expertise, then you're doing it wrong. I'm guessing these orgs think devs has some decision making influence or perhaps it looks good for recruiting? Or maybe they just want that HN juice. What next, I wonder? Neurosurgery for devs?
Also, in general, it's very useful to know the platform your software is going to be deployed to, so that you can design the software with those constraints in mind.
After you have the money for devops, sure, seems great, many benefits. Or if you are just doing it for fun and want to learn, also fine. Or if you have some requirement (health data or something) then great.
But that 20~ has to be the lowest leverage thing possible to spend that time and money on if you are just a regular SaaS, marketplace, <name your startup type> start up. I am genuinely interested how I would get a better return on that rather than fixing bugs, listening to customers, building features, etc.
The only thing I ever hear is "In 5 years you might be locked in and you will regret it." When, in reality, if most startups lasted 5 years, they would be ecstatic...
Used to work somewhere that just used some VPSes instead of managed hosting. That 20% time was spent on debugging OS issues, disk space issues, OS package updates, downtime needed to increase instance sizing, documenting (read: not actually documenting just leaving it up to the next person) how to rebuild the environment etc etc.
You can have your developers manually copy paste files to a server, which is great while it works, but it's far better to have an automated process. At that point it's not really much more work, in fact I'd argue it's less, to pick docker + kubernetes than any of the alternatives.
The real problem is k8s and "cloud native" are designed for Google's scale, not yours.
Sure, you could argue that you shouldn't expose backend services like that without an API endpoint at all, because one application->one namespace, or you could use a service mesh/consul to expose it to the things that need it, but it's an entirely different mindset from "I have a postgres cluster in public/private cloud"
I don't think that would cause issues with long lived connections. It is about as direct as you can get while actually load balancing.
https://cloud.google.com/kubernetes-engine/docs/how-to/servi...
They typical "web app" flow is Client -> ReverseProxy -> StatelessApp -> DatabaseWithState and back. The typically expectation is HTTP requests that aren't "long lived". Even running a database would be harder since you need to consider where the storage of the content is going (you'd want backups and snapshots etc) and how it'd get there. If the StatelessApp goes down (or you need more), you just add containers.
One example that wouldn't work well on k8 that I'm sure isn't very common is like a Minecraft server. It is a persistent TCP connection to a server that reads writes data to the FileSystem.
Something like order vps, apt install that, go to ip:4321, login, enter a master ip, and now it’s part of a farm to which you can drag and drop containers (or repositories, or even just init scripts with tarballs) and edit configs/secrets accordingly, and see logs.
I understand docker and what k8s does, but in our situation setting them up and managing is time-consuming overkill that brings nothing except complexity and need for yet another expensive role.
PaaS are pretty much turnkey for many apps, and cost mostly scales with the actual usage which tends to fit well small companies.
They completely hide infra management from you, deployment can be entirely automated through Git with no SSH or Docker registry management, and they provide access to what you listed: config/secrets, logs.
So if you are looking for the easy solution, use a SaaS / PaaS that really is turnkey and, for example is not just another peaky abstraction of kubernetes. You are choosing an easy solution so you should not need to invest a lot of your energy. Likewise recognize that when you need more customization that you might need to jump ship instead of investing in ducktaping your SaaS/PaaS.
~HN, circa 2010
Nomad is open source; why clone it?