Hacker News Reader

Comment by dasl | Hacker News Reader

Hi, I'm the author of the article! Thank you for the awesome description of the various vm.dirty_* sysctls.

The problem described in my post was not _directly_ related to the kernel flushing dirty pages to disk. As such, I'm not sure that tweaking these sysctls would have made any difference.

Instead, we were seeing the kernel using too much CPU when it moved inodes from one cgroup to another. This is part of the kernel's writeback cgroup accounting logic. I believe this is a related but slightly different form of writeback problems :)

View on HN

Hey, I agree that tweaking these probably wouldn't have made much difference, but tuning/reducing the dirty_bytes could calm the writeback stampede and smooth that bump, potentially getting rid of whatever race might have been happening. Regardless, disabling the cgroup accounting there is the right thing to do, especially as you don't need it. Tbh, the main reason I wrote most of that was as background to explain the cgv1 vs v2 differences and why they're there (and because I was stuck in traffic for like 45 mins :/)

If you're ever in the mood to revisit that problem you should try disabling that discard flag and see if it makes a difference. Also, if it was me, I'd have tried setting LimitNOFILE to whatever it is in my shell and seeing if the rsync still behaved differently.

Anyway - thoroughly enjoyed your article. You should write more :)

I found that info about the `discard` behavior quite interesting. And thank you for the kind, inspiring words -- cheers!

> moved inodes from one cgroup to another

`cgroup.memory=nokmem` avoids this.

TIL, thanks for sharing. We ended up solving our problem another way by adding this `DisableControllers` stanza to the service's systemd configuration: https://gist.github.com/dasl-/87b849625846aed17f1e4841b04ecc...

I believe the kernel's cgroup writeback accounting features are enabled / disabled based on this code: https://github.com/torvalds/linux/blob/c291c9cfd76a8fb92ef3d...

Reply on news.ycombinator.com