For the love of god, stop using CPU limits on Kubernetes (updated) | Hacker News Reader

For the love of god, stop using CPU limits on Kubernetes (updated)

home.robusta.dev

47 pointsciceryadam27 comments

The argument against this is consistency. Without a limit set, you are only guarenteed up to your request's worth of cpu, but you will often be allowed to have more. This can create a false sense of security, as your application is working fine (even though it occasionally exceeds its request). Until one day, when a neighbor happens to get thirsty, and your application suddenly breaks. Limits front-load the brokenness so that it happens immediately instead of randomly.

It’s a trade-off that needs to be considered on a case-by-case basis. This is one of those cases where “one rule fits all” doesn’t work.

> Until one day, when a neighbor happens to get thirsty, and your application suddenly breaks. Limits front-load the brokenness so that it happens immediately instead of randomly.

Can you expand on this? Why should the application break? It still has its guaranteed cpu requests. If it breaks with the defined requests, shouldn't it always break?

Not op, but it would break because your CPU request wasn't high enough from the start and the problem was hidden because you had CPU to spare in the node. Once you don't have CPU to spare, the app breaks.

EDIT: the vertical pod autoscaler helps with this as it will adjust requirements on pods to make sure they don't over/under allocate.

Solution - monitor your apps for when usage is consistently over request

I've had this happen in the real world.

In this case it was a GraphQL implementation that gradually grew in size, complexity and scope. The team maintaining it never adjusted the initial resource requirements within about 2 years because it has never been a problem - until a different (much larger by total CPU allocated) service started consuming all the bursting it could get and the GraphQL service just stopped processing requests. Before any autoscaling could kick in, it went into CrashLoopBackoff with failing liveness.

You can and should catch such cases early with monitoring, but our platform team was extremely tiny, especially for how many developers it served.

This advice comes from tunnel vision and makes perfect sense if you know that you have exactly two pods running at any given time. But if you have exactly two pods, then why bother use k8s? IIRC one of the major selling points of K8s was on-demand scaling or auto scaling horizontally. Which means the number of pods you have in the cluster is dynamic.

In the context of pods dynamically spinning up and spinning down, it's bad when a pod replica can't be allocated in the cluster "predictably" but there is nothing worse than when a new pod (new deployment) fails because "Marcus the pod" drank all the water and now I have to call DevOps and wait god knows how long before they spin up a new node to guarantee a spot for the new pod.

Bin-packing is a already an np-hard problem. If you remove limits from CPU then you're adding probabilities into the mix. So, for the love of god, always use limits unless you have a very specific use case.

> I have to call DevOps and wait god knows how long before they spin up a new node to guarantee a spot for the new pod.

Cluster autoscaler. No need to call anyone.

The reason to never use CPU limits is different than those stated in the article. In short: Linux kernel SUCKS. More specifically, the "Completely Fair Scheduler" (CFS) sucks at enforcing those limits. Setting any limit at all causes CFS to waste like half of CPU cycles on enforcing it, and only the other half is available for any useful work.

Can’t you use a different scheduler on your nodes?

The double accounting has been fixed a while ago. The real issue is poor defaults (100ms period is too long) and generally folks aren’t aware that limit actually sets cfs quota

Looks to me the author hasn't run different workloads in different production clusters of any complexity. Advise is fine for a small predictable cluster but too simplistic for any real complex cluster.

We run hundreds of nodes (autoscaling cluster) across multiple clusters, with wildly varying workloads, and almost none of the apps have CPU limits set. No issues.

Same here. We have eliminated all limits entirely. Same scale (maybe slightly larger). Diverse workloads. In principle we can bring cfs back with like 1ms period but we’d probably look into cpuset pinning first

We use CPU limits at work for the simple reason we can't autoscale deployments without having them set. An HPA will deploy a new pod each time the CPU limit has been reached for more than 30 seconds.

The whole point is to scale out, not up.

CPU utilization isn't a great metric to scale on. If the app is running at 90% CPU and all requests are being processed well within your SLIs, it's fine.

KEDA [0] lets you scale on any number of things - basically if you scrape it with Prometheus, you can scale on it.

[0] https://keda.sh

That is incorrect - hpa works off cpu request not limit (and usage).

When you do a "kubectl get hpa", the 0% and 100% rules are set according to the CPU request and limit. The current usage will be set to "<unknown>" if either is missing, and the deployment won't autoscale.

Again this is not correct - limits are not factored in at all. Even if you do set them it's important to understand this in case overcommit is setup. Here's the relevant offical documentation:

> For per-pod resource metrics (like CPU), the controller fetches the metrics from the resource metrics API for each Pod targeted by the HorizontalPodAutoscaler. Then, if a target utilization value is set, the controller calculates the utilization value as a percentage of the equivalent resource request on the containers in each Pod.

> ...

> Please note that if some of the Pod's containers do not have the relevant resource request set, CPU utilization for the Pod will not be defined and the autoscaler will not take any action for that metric.

https://kubernetes.io/docs/tasks/run-application/horizontal-...

Edit: here's the code: https://github.com/kubernetes/kubernetes/blob/3ffdfbe286ebce...

I don’t agree on the recommendation for memory « Always set your memory requests equal to your limits »

you can layer high priority service and low priority service better if you use some buffer.

My current model for memory on k8s tends to agree with the article.

Would it be possible for you to explain "you can layer high priority service and low priority service better if you use some buffer" further?

I mean think of low priority service as services that are not latency sensitive (background jobs).

you want those low priory cgroup to use the extra memory if it’s available (not used by a high priority cgroup)

it a high priority cgroup need the memory it steal it from the low priority cgroup up to the minimum guaranteed to this low priority cgroup.

this low priority cgroup memory contention will turns into IO pressure from page faults. Then IO limit on the low priority cgroup will cap how much IO it can generate by throttling it.

Meanwhile the high priority cgroup use it’s guaranteed cpu, guaranteed IO and guaranteed memory with no hiccup.

to learn more check « fbtax2 memory controller configuration » at https://facebookmicrosites.github.io/cgroup2/docs/memory-con...

In Linux kernel, Different memory cgroups have distinct lruvecs, for memcg reclaim.

This mean global reclaim algorithm can look at all cgroup "memory.low", and reclaim first from cgroups that are using more than their "memory.low" config.

You need to make sure that the sum of all cgroup "memory.low" is still below total memory available. Leaving 20% buffer for when the whole system is under stress is a good rule of thumb.

In addition, any memory that's guaranteed but unused can be allocated to other processes, further optimizing memory utilization.

The problem with that approach (overcommit) in k8s is when oom killer kicks in your containers will die immediately and thus you can easily corrupt some internal state if not careful. Also it’s easy to accidentally kill wrong process group bc oomkiller will calculate priority based on memory usage among other things so outsized processes can be killed before lower priority tiny processes

I don't think this is a remotely compelling argument to never use limits.

The summary should be to use a request which satisfies the long term minimum cpu required at any given time. Then if you’re happy to allow it to burst then don’t set a limit. If you want consistency and constant runtime, set a limit. If you find that your container only works when it uses the “free” burst time, and breaks when it goes back to the request cpu, then your request is too low.

good summary!

Reply on news.ycombinator.com