Linux kernel cgroups writeback high CPU troubleshooting

The article solves the cpu spike by disabling the io or memory controller, but if one would like to use those controllers, a better way to charge memory would be nice.

It is unfortunate that it's clear where the memory should be charged, but the kernel does not provide reliable way to deterministically charge that memory. If anyone has any design ideas, please feel free to chime in!

This is really not great in situations where you're bootstrapping a fresh server. Here's what happens:

- you boot up a server with, say, 1tb RAM

- your default dirty ratio is 10 (best case)

- you quickly write 90gb of files to your server (images, whatever)

- you get mad unblocked throughput as the page cache fills up & the kernel hasn't even tried flushing anything to disk yet

- your application starts, takes up 9gb memory

- starts to serve requests, writes another 1gb of mem mapped cache

- the kernel starts to flush, realises disk is slower than it thought, starts to trip over itself and aggressively throttle IO until it can catch up

- your app is now IO bound while the kernel thrashes around for a bit

This can be tuned by adjusting the vm.dirty_* defaults, and is well worth doing IMO. The defaults that kernels still ship with are from a long time ago when we didn't have this much memory available.

My memory of this next bit is flaky at best, so happy to be corrected here, but I remember this also being a big problem with k8s. With cgroups v1, a node would get added to your cluster and a pod would get scheduled there. The pod would be limited to, say, 4gb memory - way more memory than it actually uses - but it would have a lot of IO operations. Because the node still had a ton of free memory, way below its default dirty writeback ratio/bytes, none of the IO operations would get flushed to disk for ages, but the dirty pages in the page cache would still be counted towards that pod's memory usage even though they weren't 'real' memory, but something completely out of the control of the pod (or kubernetes, really). Before you knew it, bOOM. Pod oomkilled for seemingly no reason, and no way to do anything about it. I remember some issues where people skirted around it by looking off into middle distance and saying the usual things about k8s not being for stateful workloads, but it was really lame and really not talked about enough.

This might seem unrelated, but you guessed it, it was fixed in cgroups v2, and I imagine that the fix for that problem either directly or indirectly explains why OP saw a difference in behaviour between cgroups v1 and v2.

Also, slightly related, I remember discovering a while back that for workloads like this where you've got a high turnover of files & processes, having the `discard` (trim) flag set on your ssd mount could really mess you up (definitely in ext4, not sure about xfs). It would prevent the page cache from evicting pages of deleted files without forcing writeback first, which is obviously the opposite of what it was designed to do (protect/trim the ssds). Not to mention cause all sorts of zombifications when terminated processes still had memmapped files that hadn't been flushed to disk, etc.

AFAIK it's still a problem, though it's been years since I profiled this stuff. At peak load with io-intensive workloads, you could end up with SSDs making your app run slower. Try remounting without the `discard` flag (and periodically fstrim manually), or use `discard=async`, and see what difference it makes.

(3.10 released 2013-06-30)