A few days ago, I got a complaint about slow ops in a Ceph cluster, and one of the Ceph OSDs spent quite a visible amount of time in response to a simple "ceph tell osd.13 ops" query that is a natural first step when debugging slow ops. Upon investigation, it was found that one of the nodes had a significantly higher load average (50) vs others (14-18). In "top", out of 256 GB of RAM that the node had, approximately 100 GB were cached and thus formally available, yet, out of 8 GB of swap, 6 GB were used, and kswapd consumed a non-negligible percentage of the CPU time.
The kernel is some 5.15.x Ubuntu kernel; the root disk (which also holds the swap file) is a cheap SATADOM, and SATADOMs, in general (yes I know there are exceptions), have miserable performance and low write endurance; I forgot to check SMART statistics but cannot exclude that the drive might have started failing. I did check the dmesg, though, and there were no I/O errors.
The problem has been resolved by disabling the swap file. I should probably have set up swap on ZRAM instead, but a configuration without swap also works.