Linux Kernel 5.14
lwn.net
lwn.net
A better source for these changes are the two merge window roundups from LWN:
Memory allocated with memfd_secret gets removed from the direct map, which means that from any other process's address space, there is no virtual address (regardless of memory protection) that corresponds to those physical addresses. So any exploit that gets only gets you memory read/write but not kernelspace code execution (various types of wild pointers, etc., as well as whatever exciting speculative execution vulnerability will undoubtedly show up next year) cannot compromise the data in a memfd_secret mapping.
That's an odd assumption. There are still a lot of universal tricks for turning a truly arbitrary R/W primitive into code execution, e.g. just write to the page table.
OTOH, this does guard against arbitrary read primitives, which, as you mentioned, is especially useful in post-Spectre days.
e.g. if you had a sidechannel that leaks data from the kernel, the kernel can't disclose data that it can't actually read.
Kubernetes implements CPU and memory limits for containers (https://kubernetes.io/docs/concepts/configuration/manage-res...). The memory limit works the way you'd expect: there's a memory cgroup, and Linux implements the memory-allocation algorithm based on the memory cgroup's limit and not the total system memory. Wildly-overboard allocations get denied; somewhat-overboard allocations are permitted if you have overcommit turned on, but cause an OOM kill if you try to use all of it. This is at least expected behavior for existing non-Kubernetes applications and (more importantly) for people used to non-Kubernetes application development/deployment: it's just like if you were running on a Linux system with the limited memory.
CPU limits are much wackier. Because CPUs are a discrete resource and Kubernetes supports fractional CPU allocations, they can't use CPU pinning. Instead, they use the CFS quota controls, which aren't commonly used outside Kubernetes as far as I can tell. The quota is expressed as basically a fraction: for a process with, say, 0.5 vCPUs allocated, it gets to use 50 ms of CPU time every 100 ms of real time. The way it works at the kernel level, as best as I've been able to understand, is that after a process runs on the CPU for a bit, the kernel looks at the quota and sees if it has exceeded the quota. If it has, the process becomes unschedulable until it has "re-earned" sufficient quota. It does not slow down or proactively de-schedule the process, analogous to how it would have run on a slower or fewer-CPU computer.
So, if you have multiple processes or threads running in your container, and there are free cores for them to run, you can easily use up multiple seconds of CPU time in those 100 ms. Then you have to wait to re-earn your quota in 50 ms chunks before your process can run. And all the while the process appears to be runnable, not stuck, it just seems like the kernel is never choosing to schedule it.
We have seen, in production, processes stalled for minutes because they evidently ran so far past their quota. We have seen entire system deadlocks when such a stalled process is holding some kernel lock for network I/O, in our case, Ceph (if the kernel releases the process into interruptible sleep, then that process can still hold locks; it will release those locks if signaled and rescheduled, but in this case the process isn't being scheduled at all).
And we're far from the only people who have seen awful behavior from CFS quotas. Tim Hockin, Kubernetes co-creator and tech lead, advises people to just never use the CPU limit feature! https://twitter.com/thockin/status/1134193838841401345 We have done as he advises and just turned it off cluster-wide with --cpu-cfs-quota=false - it helps the Kubernetes scheduler if you set your CPU/memory limits equal to your requests, so this gets us the right Kubernetes behavior without incurring the wacky kernel behavior. If you Google that flag you can find many others who have done the same in their public configurations. There's a Kubernetes feature to use CPU pinning instead, which mentions improved performance as an advantage https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-... , but it only works if you're scheduling whole numbers of CPUs always.
A whole lot of applications on our Kubernetes clusters - lightly-accessed web servers, monitoring daemons, etc. - are best described as wanting 0.1 or 0.5 CPUs, in the sense that on (say) an 8-core machine, you could run many dozens of instances of these applications just fine. But that's not how the existing CFS quota mechanism interprets 0.1 or 0.5 CPUs. It seems like the "burst" mechanism here is likely to handle processes that, in a random 100 ms period, actually do run for more than 10 ms, but on an ongoing basis don't average out to using more than one tenth of a CPU. We might be able to turn --cpu-cfs-quota back on as a result.
Bad, bad, bad scenario all around. A gameserver process really needs to get scheduled like a heartbeat. Ticks can't be late on the server as it really screws everything else up big time. We've seen these "bursting" effects really start to take it's toll, to the effect that we're now just setting a request and limit of 1CPU for each game server process (even though we could pack a lot more punch here).
I think you need a core. If you don’t have one you’ll never know when the scheduler runs the process. The core scheduler feature seems like a good way to now do this.
We used to run hpux in “semi real time mode” for processes on tight timelines. The OS would let us put one process on the core that had interrupts disabled (only certain system calls were allowed). It worked well. Of course if the process had an issue the machine required a reboot.
Hpux also had processor sets which would allow us to tell the os to run certain processes on certain cores. This seems similar to the “core cookie” but I think that seems to have more to do with resource control.
This is really good progress.
It has worked surprisingly well, I can compile Firefox on all cores while playing a game and it's completely imperceptible that anything else is happening, both in terms of performance and interactiveness/latency. No issues with this "burstiness" or stuck/deadlocked processes, either. "Nice" is basically useless in comparison.
The problem is that we're running services etc., not batch jobs, so we do want them to make meaningful forward progress. So we can't set the shares to zero. We just don't want a misconfigured / runaway job to starve out the rest of the machine, even when the other jobs are not trying to use 100% CPU.
A specific sub-case here is capacity planning - right now, even with CPU shares, you can request one CPU for a computationally-intensive multi-thread/process task, and if the rest of the box is running internal web services with sporadic traffic, it will easily be able to use the whole machine. But then if you launch eight instances of that same job on the machine, they'll all perform much worse. So ideally we want to proactively limit CPU usage so that application developers/operators get realistic expectations about performance, and in turn, we get realistic information about how heavily our cluster is actually used.
... So let me ask you this, by when do you get this kernel available to you in production? ;)