I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpus" allocated to the job, but upon further reflection, that probably wouldn't work for many reasons.
I wondered for a while if docker could make a fake /proc/cpuinfo that apps could parse that just listed "docker cpus" allocated to the job, but upon further reflection, that probably wouldn't work for many reasons.
(There are exceptions, such as documented here: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...)
flask/gunicorn will fork 16 processes (by reading /proc/cpuinfo and counting cores) all of which will try to share 0.5 cores worth of CPU power (maybe spread over many physical CPUs; I don't really care about that).
I can solve this by passing a flag to my application; my complaint is more that apps shouldn't consult /proc/cpuinfo, but have another standard interface to ask "what should I set my max parallelism (NOT CONCURRENCY, ROB) so my worker threads get adequate CPU time so the framework doesn't time out on startup.
Been a bit but I do believe that dotnet does this exact behavior. Sounds like gunicorn needs a pr to mimic, if they want to replicate this.
Caveats: up to the number of cores exposed to your VM. I also believe the later versions of K8s let you do some degree of workload-core pinning and I don’t yet know how that interacts with core availability .
If you set cpus to 2 and another process sets to 1 and you both try to use all CPUs all out, you’ll get 66% and they’ll get 33%.
This isn’t the same as cpusets, which work differently.
That's not true at all. Take a look at `cpu.cfs_quota_us` in https://kernel.googlesource.com/pub/scm/linux/kernel/git/glo...
It's a hard time limit. It doesn't care about contention at all.
`cpu.shares` is relative, for choosing which process gets scheduled, and how often, but the CFS quota is a hard limit on runtime.
Not at my work environment right now, don't remember the parameters I used.
docker run --cpus 0.1 --rm -it progrium/stress --cpu 16
and the machine's cpu sits idle.The true objective function I'm trying to optimize isn't just "save money" or "don't waste CPU resources", but rather "get a million different workloads to run smoothly on a large collection of resources, ensuring that revenue-critical jobs always can run, while any spare capacity is available for experimenters, up to some predefined limits determined by power capacity, staying within the overall budget, and not pissing off any really powerful users." (well, that's really just a simplified approximation)
At work at my new job we run mixed workloads in k8s and I don't really see a problem, but we also don't instrument well enough that I could say for sure. In our case it usually just makes sense to not oversubscribe machines (Google oversubscribed and then paid a cost due to preemptions and random job failures that got masked over by retries) by getting more machines.
I know the JVM automatically uses it, and there's a popular library for Go that sets GONAXPROCS using it.
> Print the number of processing units available to the current process, which may be less than the number of online processors. If this information is not accessible, then print the number of processors installed
https://www.gnu.org/software/coreutils/manual/html_node/npro...
https://www.flamingspork.com/blog/2020/11/25/why-you-should-...
if [[ -e /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]] && [[ -e /sys/fs/cgroup/cpu/cpu.cfs_period_us ]]; then
GOMAXPROCS=$(perl -e 'use POSIX; printf "%d\n", ceil($ARGV[0] / $ARGV[1])' "$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us)" "$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us)")
else
GOMAXPROCS=$(nproc)
fi
export GOMAXPROCS
This follows from how `docker --cpus` works (https://docs.docker.com/config/containers/resource_constrain...), as well as https://stackoverflow.com/a/65554131/207384 to get the /sys paths to read from.Or use https://github.com/uber-go/automaxprocs, which is very comprehensive, but is a bunch of code for what should be a simple task.
Just setting it to 16 will satisfy 99% of users.
Fundamentally, intelligent apps need to interrogate their environment to make concurrency decisions. But I agree- Go would probably work best if it just picked a standard parallelism constant like 16 and just let users know that can be tuned if they have additional context.
Most people are running on systems where the CPU capacity varies and they haven't even noticed. For example in EC2 there are 8 victim CPUs that handle all the network interrupts, so if you have an instance type with 32 CPUs, you already have 24 that are faster than the others. Practically nobody even notices this effect.
Fascinating. Could you share any (all) more detail on this that you know? Is it a specific instance type, only ones that use nitro? (or only ones without?) This might be related to a problem I've seen in the wild but never tracked down...
This is new to me. What is this… behavior? What keywords should I use to find any details about it?
The only thing that rings a bell is requests/limit parameters of a pod but you can't change them on an existing pod AFAIK.
What I tried: I created a "Burstable" Pod and run `nproc` [0] on it. It returned N CPUs (N > 1).
Then I created a "Guaranteed QoS" Pod with both requests and limit set to 1 CPU. `nproc` returned N CPUs on it.
I went back to the "Burstable" Pod. It returned N.
I created a fresh "Burstable" Pod and run `nproc` on it, got N again. Please note that the "Guaranteed QoS" Pod is still running.
> Pods with Guaranteed QoS will have exactly the number of CPUs they asked for, no more or less
Well, in my case I asked for 1 CPU and got more, i.e. N CPUs.
Also, please note that Pods might ask for fractional CPUs.
[0]: coreutils `nproc` program uses `sched_getaffinity` syscall under the hood, at least on my system. I've just checked it with `strace` to be sure.
FWIW, `taskset` uses the same syscall as `nproc` (according to `strace`).
# taskset -c -p 1
pid 1's current affinity list: 1
# nproc
1
I honestly do not see how it can work otherwise.In my case it's `"cpuManagerPolicy": "none"` and I suppose you're using `"static"` policy.
Well, TIL. Thanks!
This exists: https://github.com/lxc/lxcfs
lxcfs is a FUSE filesystem that mocks /proc by inferring cgroup values in a way that makes other applications and libraries work without having to care about whether it runs in a container (to the best of its ability - there are definitely caveats).
One such example is that /proc/uptime should reflect the uptime of the container, not the host; additionally /proc/cpuinfo reflects the number of CPUs as a combination of cpu.max and cpuset.cpus (whichever the lower bound is).
As others also mentioned, inferring the number of CPUs could also be done using the sched_getaffinity syscall - this doesn't depend on /proc/cpuinfo, so depending on the library you're using you might be in a pickle.
Oh they did, they're a modern day IBM.
> Containers are a crappy abstraction
They're one of the best abstractions we have (so far) because they contain only the application and what it needs.
Delusion level: over 9900.
I'm yet to find a container that contains only the application and what it needs. Most of the time I find that they contain at least libc and libpthreads (which are already present on the host, so not needed). More often I find metric tonnes of garbage that was not necessary by any metric, but was just too hard to remove, so was allowed to stay.