Comment by mikehollinger | Hacker News Reader

Comment by mikehollinger | Hacker News Reader

We use kubernetes to spin up the application that I work on (in private cloud and at some point in hybrid and public cloud) deployments. It’s an end user installed tool. In deployment, about 1/4 of the new installations fail because of some problem or another. Either the GPU plugins for NVIDIA weren’t loaded correctly, kube-dns won’t start because docker0 isn’t in a “trusted” zone in redhat (not being in trusted seems to cause iptables to subtly screw up container to container communication between the various private networks), or helm just decides that it can’t start.

Are we doing it wrong?

We’re using hyperkube and k8s 1.8 which came out around q4 of last year.

Almost all of these I can trace back to user error (ie we told folks to do X, they didn’t, and stuff broke). We’re now having to write a preflight checklist of sorts that the app runs through to make sure A bunch of stuff is “ok.” That in itself becomes brittle in my experience so I’m reluctant to do that.

It requires considerable operational experience and effort to run well on bare metal. Have you considered experimenting with a managed Kubernetes offering? Out of EKS, AKS, and GKE, I cannot recommend GKE highly enough.

Even if you can’t use it for production, it’s highly worth setting up a prototype environment on GKE to see what it gives you. I believe they now have GPU support, including support for preemptible GPUs (much cheaper).

Also GKE does a good job of staying up to date with releases. They are on 1.10, with support expected soon for 1.11. They _fully_ manage the upgrade of both the Kubernetes and etcd masters, and the worker nodes.

As mentioned in the linked blog post, Kubernetes is a fast moving project, and to use it you should plan and allocate significant resources in your team to keeping up with the new releases. There are a large number of fixes and improvements since 1.8 and I would look very seriously at both upgrading, and changing your processes to allow you to stay closer to the current release version.

The Kubernetes project does not, and has no current plans to, have a long term support release.

You pretty much have to maintain an install script for bare metal. It has to check everything and install&configure everything.

We then ask to rerun the script. If it fails, we fix the script.

What is now happening around Kubernetes is that various companies and projects are coming up with "distributions" of their own. These package the underlying OS, the Kubernetes binaries, and a number of built-in system pods that take care of various things such as logging, scheduling and monitoring. Red Hat's OpenShift is one of them, as well Weaveworks. Maybe try having a look at one such solution?

There is also the Canonical Distribution of Kubernetes (CDK).

If you want to try a stripped down version on your local machine, kubernetes-core can be installed into LXD containers via conjure-up with a click of the fingers - nice for playing around.

Not sure if I got the scenario. Are you installing K8s on bare metal using a previously installed OS that you do not control?

If so, that will be harsh. I am not sure how to help you there. There are many things that could go wrong as you say.

If you do have control over the OS (you are providing an ISO or virtualization image) then it should mostly "just work". My company is doing a similar thing, only we ship boxes with everything pre-installed. There is another scenario for hybrid cloud, but even then they download a VMWare image.

Also if you do have control over the OS: can you use CoreOS instead? It is very well suited for running K8s and has less things that can go wrong. RedHat bought them anyway. With Ignition (or even the old-fashioned cloud init with a config drive), it is a no-touch deployment (you do have to generate and inject the certificates beforehand).

One thing that sounds weird is that you are "telling folks to do X". Can you avoid telling them anything and have it automated?

actually you should probably look into kubeadm or bootkube. Currently I tried to have some kind of scratch cluster myself, however it's just way easier to maintain/update clusters that are on kubeadm or bootkube.

also stuff that you describe does not look like some user error. btw. you should not expose the deployment yaml/json to "users"/"developers". You should have a ci that just runs `kubectl set image deployment/name pod-name=IMAGE` and keep all deployment descriptors, etc in a seperate source repository.

Isn't this exactly what you should expect from kubernetes? If you can't afford a team of people working on it you are supposed to give up and accept GKE lock-in.

I don't think GKE is lockin when there are other offering from other companies and the migration is quite simple. Also if at some point you feel like it's time to host your own then you can do it. Also I don't find GKE that expensive especially paired with their cheap vms or preemptive vms if you're fine with those.

Reply on news.ycombinator.com