Hacker News Reader

Comment by YawningAngel | Hacker News Reader

I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.

View on HN

This post explains it - https://www.reddit.com/r/kubernetes/comments/all1vg/comment/...

Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.

I'm not in a good position to say for sure, as I've only used managed Kubernetes distributions, but I think it probably works out to less work than removing CPU limits. Kernel upgrades are at least semi-routine, so most shops that run Kubernetes themselves are going to have a process for them. Conversely, removing CPU limits and migrating the critical path to a different set of tainted nodes is a substantial one-off change with a long tail of failure scenarios that need to be tested. Thus, I would expect that a kernel upgrade would be easier than doing what Buffer did.

It should be. Kernel updates are a critical path for bug fixes and security patches. If you can’t upgrade your kernel using a clear, premeditated plan, you are not doing your job right.

By and large, you can just upgrade the kernel and nothing will break. It's one of the biggest things kernel developers worry over, in fact.

The reason, I think, K8s people might do this is that they often do not have the access, experience, or skills to do platform upgrades. It's easier to just futz around on the layer they feel they understand well; the world of containers.

You can test out your shit by creating a new node with the recent kernel version installed. Then schedule some apps for testing on that node and see if everything works. Swap it out when you're done testing.

Can’t remember the last time I had a centos kernel update (kernel-ml) break anything over the past 7 odd years I’ve been running it across hundreds of servers.

Not less straightforward than any other system upgrade, which you need to be able to do confidently, anyway.

It's a single command that is safe and straight forward especially if you use a LTS distro.

Yes, you just install the kernel image, kernel headers and kernel modules and reboot. You are not upgrading the distro. With Kubernetes it would be quite easy to test in isolation by simply tainting the worker node(s) that have the upgraded kernel.

Reply on news.ycombinator.com