For the Love of God, Stop Using CPU Limits on Kubernetes | Hacker News Reader

For the Love of God, Stop Using CPU Limits on Kubernetes

home.robusta.dev

22 pointsarguflow23 comments

I used to be on the same page as the author. But then I saw tons of application teams not setting CPU limits, and coming to rely on the bursting (in other words, their requests were too low). Thus when the system came under load their application started slowing in unexpected ways.

We've had success with CPU limits, and horizontal scaling.

He does address that at the end of the article with the dashboard that looks at historical usage to inform your cpu requests.

It's sad to me that Kubernetes doesn't expose the excellent hierarchical system for rationing CPU that's built into the kernel: cgroups. It has its own separate constraint system, makes its own scheduler. And it just seems not as good, not as flexible, as the hierarchical system cgroups offers.

Being able to make a cgroup where essential services as a whole share a pool guaranteed 30%, then further refining & trading off that pool & other work pools feels like such a superpower. Compared to having to manage all services in flat, absolute terms.

I’d say being able to set I/O limits can be much more useful than CPU limits regardless of platform. Less chance of bringing an entire host to a halt.

Can't your operating system manage your CPU resources for you already? Why does Kubernetes need to be involved in process scheduling?

Kubernetes is giving my operating system the cpu to manage.

Almost 2-yo post and still as wrong as it was when it was posted.

What’s wrong about it? I haven’t managed k8s much and don’t have enough experience to evaluate their claim.

I can't comment about about other statement.

But there are serious drawbacks if you don't set requests/limits for mission-critical process, is that they can be killed by kernel to free some resources (if the node reach max resource usage)

When you don't set cpu/memory limit your pod QoS class is burstable, which better then BestEffort, but still get assigned `oom_score_adj` score. IMHO you almost 99% you want `Guaranteed` for critical process.

1. oom_score_adj - https://kubernetes.io/docs/concepts/scheduling-eviction/node...

2. Guaranteed - https://kubernetes.io/docs/tasks/configure-pod-container/qua...

3. QoS - https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/

The post recommends setting both request and limit for memory. It's only for CPU that it says to use a request and not a limit. It's my understanding that the kernel only kills processes when it's out of memory. In the docs you linked, there doesn't seem to be an "eviction signal" for CPU.

This doens't match what I've read.

IF you don't set limit to cpu & memory, your pod QoS will be burstable and from kernel/cgroup perspective this process can be killed - it's only question of oom_score_adj.

OOMScore from linux - https://www.freedesktop.org/software/systemd/man/latest/syst...

OOM means "Out Of Memory". The post is about CPU.

> Everything in this post is about CPU and not memory.

I am aware of it, are you aware that Kubernetes assigned Burstable Qos if you don't defined CPU requests and limits?

What I'm trying to say that while try to avoid one pitfall you fall right into an other - in short there are good reasons to use cpu limits.

That's not what you're saying though. You're also putting requests/limits for CPU and memory in the same class, which they're not.

Lack of CPU will never cause a pod to be killed. Evicted, yes. Killed, no.

Lack of memory can cause a pod to be OOM killed. That's an entirely different failure mode.

Eviction should always be expected by your pods. They should be able to handle eviction gracefully. On the other hand, there is nothing you can do to handle an OOM kill gracefully.

K8s works best (economically) when it can bin pack things. The only way it can bin pack things safely is by having user provided limits--its not smart enough to right size your app (out of the box at least). Not setting them means you're going to end up paying more or have resource contention/outages.

It's better to set the limits higher than you need than to not set them at all. Ideally this is easily done since you're profiling/load testing your app and you understand the appropriate sizing for it, right?

The problem is that the Kubernetes CPU scheduler is... not great.

There are many instances where it will throttle CPU of a pod way before it even reaches the requested amount, let alone the limits. This is only true if you set limits. If you don't, it will always have the requested amount available, plus whatever can be spared that's not in use by other processes. It will throttle at most down to the requested amount, and in most cases not at all.

Unless your app can use all the cpu you assign it

Ha! Always end with a joke, I love it.

Users come to expect the performance that was never promised or even properly requested. Once you efficiently load the system they complain because you overdelivered and they got used to it.

We've been fixing that over time and now no one expects a computer to be as fast as it accidentally was in 2010.

Why, by 2030, computers ought to be even slower!

I'm sorry, in what way are modern servers not fast enough to serve a few hundred CRUD requests per second?

Yeah, it's a pretty weird take. I would agree if they said "a lot of people don't know what CPU limits are and set them without thinking", but "you should never use them" doesn't make any sense. For example, every computer anyone has ever used has a CPU limit. There isn't infinite CPU inside your computer. That's a CPU limit.

The mistake that surprises people is "I'm going to tell my app it can use 128 CPUs but I'm setting the CPU limit to 1." OK, well, then your app is going to be asleep 127/128th of the time. It's a surprise because the app reads /proc/cpuinfo to guess how many CPUs you have, but ... that is not the correct algorithm.

Another thing is imagining that usage spikes are going to occur randomly throughout time. If app A is under heavy use, it can steal app B's CPU shares, because who would use app A and app B at the same time? Most of the time they're both idle, so it's a waste to reserve 1 CPU for app A and 1 CPU for app B, and have app A throttled while app B is idle. But you'll probably find that everything you host is popular from 9am to 5pm local time, and for 16 hours a day you are using 0% CPU and for 8 hours a day you are using 200% CPU. The idea is to guarantee some quality of service for both apps, even at busy times. The goal is not to maximize overall throughput.

You can tune your latency vs. throughput goals if all the apps are yours, but as soon as you have different teams, I doubt team B is going to say "sure we can get paged for high request latency as long as Team A is getting as much of the CPU as they can". That's what CPU limits are for, consistency when things get tough. Not for overall utilization.

Reply on news.ycombinator.com