Thoughts on Kubernetes
blog.nelhage.com
blog.nelhage.com
The docs seem to expect that:
- If you're not letting managed services do most of the work for you, you're running on one of CoreOS, Ubuntu or a RHEL derivative
- You know all about "CNI" and how it magically makes everything work
- You know that the default "security" is none at all
Deployment is just so much harder than it should be. Fundamentally (I discovered far later on in the process), Kubernetes is comprised of roughly the following services: kube-apiserver, kubelet, kube-proxy, kube-scheduler, kube-controller-manager. The other dependencies are: A CA infrastructure for certificate based authentication, etcd, a container runtime (rkt or Docker) and CNI.
The steps for setting up a simple installation should be able to fit on a single page. If it weren't for a bug crippling my cluster, I would have done it myself (it seemed to be running but a bug with cadvisor's disk space detection messed it up to the point where it was useless).
To get a handle on what I mean, have a look at the docs for Ceph, another reasonably complex distributed system. Here are the manual installation docs: http://docs.ceph.com/docs/master/install/manual-deployment/. It's a list of commands to run and a basic idea of what the config files should look like. Following that guide actually results in a running Ceph cluster. Now have a look at the manual installation docs for Kubernetes: https://kubernetes.io/docs/getting-started-guides/scratch/. It's a bunch of links to other parts of the docs, pretty much no practical guidance on how to actually set up the cluster and the few commands that have been suggested are mostly outdated (for example "--configure-cbr0" doesn't exist on kubelet anymore). Following it can result in a working cluster but only with a lot of additional work and study (it doesn't give you anything on networking, which is essential).
There are a few very nasty pitfalls on the way to your k8s install in production though...
We deploy kubernetes on hundreds of nodes in multiple locations and i've to say i've never ever looked at any install docs. ACK, they are basically useless, but overall i find the documentation quite good (except not up to date with the code on may locations).
What's severly missing is the overall picture!
But lets not stop there, if you want to have k8s running properly in production you practically have to make all containers yourself, otherwise some shitty container from dockerhub will bring your node down one after another (hint: read-only containers).
A deployment redeploys (i.e. creates a new ReplicaSet and rolling-updates to it) whenever its YAML changes. So the common way to do a "no-op" redeploy (assuming `imagePullPolicy: Always` on the containers) is to add an annotation to the PodSpec that has no actual meaning, but which you can modify to trigger the creation of and migration to a new ReplicaSet. For example, you could put a timestamp, or you just count it up every time you re-deploy.
I would discourage usage of unchanging image tags, however. When you have immutable image versions, you will have a much easier time rolling back to a previous release if the new release breaks. At $work, we use human-readable timestamps most of the time, e.g. `201702210945`, that signify when the image was built.
> The final question I don’t have a satisfying answer for is how to deploy configuration data into my docker images.
Did the author look at https://github.com/kubernetes/helm?
> Kubernetes secrets are stored in plaintext in etcd, which is fine for many applications but would probably scare me at a certain scale.
Yeah, that's indeed a problem. The best you can do right now is wrap access to Kubernetes and etcd in as many layers of firewall and two-factored SSH and whatnot as you feel are necessary.
NOTE: Answer was updated multiple times while I read through the submission.
As we run > 500 instances on AWS and GCP combined, we gave K8S a try, to see if we can get a shorter+simpler+cheaper way for our production.
K8S on GKE (Google Container Engine) turned out not to be a good fit for our needs at the moment. Instead, we found ways to get more efficient with platfrom's provided provisioning API to gain all what we wanted with K8S in a far mature env.
Wait, isn't the layering one of the whole points of Docker? You build that base image then use that tag in the other three docker files that'll then build on top of that.
The author does then mention issues with tagging, and I can relate to that more. Our team ended up with a cludgy mess of tags and we all had some form of a docker rmi command aliased for quick use every now and then. The build server had a tough time of its own
Although it's great as an open source project - it's fun to finally have a nice distributed computing platform that works anywhere. Docker swarm is great too.