Hacker News Reader

Comment by dekhn | Hacker News Reader

Maybe I should be clearer: Let's say I have a 16 core host and I start a flask container with cpu=0.5 that forks and has a heavy post-fork initializer.

flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that).

I can solve this by passing a flag to my application; my complaint is more that apps shouldn't consult /proc/cpuinfo, but have another standard interface to ask "what should I set my max parallelism (NOT CONCURRENCY, ROB) so my worker threads get adequate CPU time so the framework doesn't time out on startup.

View on HN

It's not clear to me what the max parallelism should actually be on a container with a CPU limit of .5. To my understanding that limits CPU time the container can use within a certain time interval, but doesn't actually limit the parallel processes an application can run. In other words that container with .5 on the CPU limit can indeed use all 16 physical cores of that machine. It'll just burn through it's budget 16x faster. If that's desirable vs limiting itself to one process is going to be highly application dependent and not something kubernetes and docker can just tell you.

It won’t burn through the budget faster by having more cores. You’re given a fixed time-slice of the whole CPU (in K8s, caveats below), whether you use all the cores or just one doesn’t particularly matter. On one hand, it would be would nice to be able to limit workloads on K8s to a subset of cores too, on the other, I can only imagine how much catastrophically complex that would make scheduling and optimisation.

Caveats: up to the number of cores exposed to your VM. I also believe the later versions of K8s let you do some degree of workload-core pinning and I don’t yet know how that interacts with core availability .

That interface partly exists. It's /sys/fs/cgroup/(cgroup here)/cpu.max

I know the JVM automatically uses it, and there's a popular library for Go that sets GONAXPROCS using it.

https://stackoverflow.com/questions/65551215/get-docker-cpu-...

Been a bit but I do believe that dotnet does this exact behavior. Sounds like gunicorn needs a pr to mimic, if they want to replicate this.

https://github.com/dotnet/runtime/issues/8485

You generally shouldn't set CPU limits. You might want to configure CPU requests which is guaranteed chunk of CPU time that container will always receive. With CPU limits you'll encounter situation when host CPU is not loaded, but your container workloaded is throttled at the same time, which is just waste of CPU resources.

According to the article, this is not true. The limits become active only when the host cpu is under pressure.

I don't think that's correct. --cpus is the same as --cpu-period which is cpu limit. You can easily check it yourself, just run docker container with --cpus set, run multi-core load there and check your activity monitor.

CFS quotas only become active under contention and even then are relative: if you’re the only thing running on the box and want all the cores but only set one cpu, you get all of them anyway.

If you set cpus to 2 and another process sets to 1 and you both try to use all CPUs all out, you’ll get 66% and they’ll get 33%.

This isn’t the same as cpusets, which work differently.

> CFS quotas only become active under contention

That's not true at all. Take a look at `cpu.cfs_quota_us` in https://kernel.googlesource.com/pub/scm/linux/kernel/git/glo...

It's a hard time limit. It doesn't care about contention at all.

`cpu.shares` is relative, for choosing which process gets scheduled, and how often, but the CFS quota is a hard limit on runtime.

Yes, there are hard limits in the CFS. I have used them for thermal reasons in the past, such that the system remained mostly idle although some threads would have had more work to do.

Not at my work environment right now, don't remember the parameters I used.

I just tried it and you are right. Ran

    docker run --cpus 0.1 --rm -it progrium/stress --cpu 16

and the machine's cpu sits idle.

It's complicated. I've worked on every kind of application in a container environment: ones that ran at ultra-low priority while declaring zero CPU request and infinite CPU limit. I ran one or a few of these on nearly every machine in Google production for over a year, and could deliver over 1M xeon cores worth of throughput for embarassingly parallel jobs. At other times, I ran jobs that asked for and used precisely all the cores on a machine (a TPU host), specifically setting limits and requests to get the most predictable behavior.

The true objective function I'm trying to optimize isn't just "save money" or "don't waste CPU resources", but rather "get a million different workloads to run smoothly on a large collection of resources, ensuring that revenue-critical jobs always can run, while any spare capacity is available for experimenters, up to some predefined limits determined by power capacity, staying within the overall budget, and not pissing off any really powerful users." (well, that's really just a simplified approximation)

The problem is your experience involves a hacked up Linux that was far more suitable for doing this than is the upstream. Upstream scheduler can't really deal with running a box hot with mixed batch and latency-sensitive workloads and intentionally abusive ones like yours ;-) That is partly why kubernetes doesn't even really try.

This. Some googlers forget there is a whole team of kernel devs in TI that are maintaining patched kernel (including patched CFS) specifically for Borg

I used Linux for mixed workloads (as in, my desktop that was being used for dev work was also running multi-core molecular dynamics jobs in the background). Not sure I agree completely that the Google linux kernel is significantly better at this.

At work at my new job we run mixed workloads in k8s and I don't really see a problem, but we also don't instrument well enough that I could say for sure. In our case it usually just makes sense to not oversubscribe machines (Google oversubscribed and then paid a cost due to preemptions and random job failures that got masked over by retries) by getting more machines.

I think you touch on the key issue which is the upstream scheduler does not have all the stats that you need to have confidence in the solution. You want to know how long threads are waiting to get on a CPU after becoming runnable.

`gunicorn --workers $(nproc)`, see my comment on the parent

Reply on news.ycombinator.com