Hacker News Reader

Comment by robotmay | Hacker News Reader

I've been playing with kubernetes for the past month and I'm just now deciding not to go with it for our new production systems, mostly because I just don't understand it well enough to know how to fix it if it goes wrong.

There's a lot of cool things about kubernetes (e.g. I had an automated SSL cert fetcher for LetsEncrypt that applied to any SSL ingress I added) but it still does some weird things sometimes (like constantly trying to schedule pods on instances without enough spare memory, and then killing other pods because of that; fairly certain that's not supposed to happen).

I think I'll revisit it next year and hope that it's a bit easier to get into. I'm especially hopeful about using it with Spinnaker and some sort of CI, though I couldn't find anything lighter weight than Jenkins that was straightforward to get set up on it.

View on HN

> constantly trying to schedule pods on instances without enough spare memory

I assume you're explicitly declaring the memory requirements for the application?

EDIT: I ask because usually I read examples just throwing deployments at a cluster, ends up the more explicit you are with the app requirements (CPU/mem, etc.) the better a job the scheduler can do. I realize this sounds like advanced common sense.

Aah yeah that might have helped. I hadn't been setting my memory requirements as I didn't technically know what they were :D

That's good to know for next time!

There's a great example on why setting limits is a good thing from one of Kelsey Hightower's talks, for some reason your initial comment triggered remembering this advice: https://youtu.be/HlAXp0-M6SY?t=21m51s

Requests/limits settings also decide what "quality of service" class a pod will use.

Setting requests and limits to the same values puts it in "Guaranteed", whereas setting requests and limits to different values gives you "Burstable". The default QoS class is "BestEffort", which makes the pod expendable. (This isn't the best-documented part of Kubernetes.) (Edit: You can see the QoS class of a pod with "kubectl describe pod <name> | grep QoS".)

QoS classes are important for scheduling. When Kubernetes needs to evict pods, it will pick burstable and best-effort pods first.

Note that setting memory limits can sometimes not work well with apps that use GC. I have had particularly disappointing experiences with Go apps. Go's GC is a bit odd in that it reserves a huge chunk of virtual memory (which is why Go apps, if you look at "vsz" only, often seem to take more RAM than expected), and is not very aggressive about releasing it. I have a Go app that uses only a few megs actual, but because of the GC will allocate half a gig and get OOM-killed by the kernel before the GC is able to collect.

You have confused BestEffort with Burstable. Agreed about the docs.

Gah, of course I did. Thanks, edited.

Jenkins and Spinnaker go hand-in-hand. The Jenkins stage is a first class citizen in a Spinnaker pipeline.

Spinnaker + K8s is an incredible combo. We actually ran it inside Kubernetes, and it was able to upgrade itself. Pretty awesome stuff.

I have the exact same issue with it. It just tries to do everything and it's too much to wrestle with when you just want the basics.

Reply on news.ycombinator.com