Comment by raffraffraff | Hacker News Reader

Comment by raffraffraff | Hacker News Reader

Parent Full thread

> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?

Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up.

If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen:

* Memory rises

* Some pages get swapped out

* Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes

* Memory rises some more

* You see more pages get swapped out

* (...and see more disk writes)

* Memory rises some more

* You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process.

* Disk queue lengths rise

* One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up)

This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk.

But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit.

So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out?

NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.

Thanks for the comment, that's very interesting. I hadn't considered the possibility of the desktop environment needing to do disk reads, I was thinking everything was in memory already.

I suppose the solution is to figure out how to reduce or eliminate the number of disk reads after the DE initially loads, or cache everything needed in memory with a "do not evict" flag on it. Especially for the TTY, that's basically a crucial system function and keeping everything needed for it always available in memory wouldn't take much space relative to a typical desktop system.

I think at least some of the problem is the program code pages getting evicted to disk, so they have to be read back to memory too, which goes to the back of the disk queue. When I'm experiencing thrashing, even basic stuff like alt-tab to switch windows doesn't work. And I assume the window manager doesn't need to do disk reads to switch windows. Even the damn X cursor freezes!

Desktops could be improved quite a bit by tweaking default nice and ionice priorities for apps, instead of relying on one scheduling algorithm to be able to just "figure it out" on desktops, servers, phones, etc. A memory eviction priority, if niceness isn't treated as memory priority, could also help, but the existing tools can already go a long way and they just aren't being used.

I was able to get glitch-free audio by tweaking software and soft-irq realtime and non-realtime priorities, when the default Ububtu (with low-latency kernel) was nearly unusable for audio.

This problem is there even if you have no swap at all, so I don't see different devices or zswap helping.

> This problem

Which problem? Yes, there's always disk contention. But if you don't have swap enabled then you won't have pathological swap activity dominating the disk.

There are several bad things that happen when you run out of memory. One of them is pathological swapping. Another is loss of disk caches. Another is (eventual) oom. The one that has the most detrimental effect, and typically results in the system having to be reset, is swap activity saturating disk io. Even when you've got fast SSDs, because they're still orders of magnitude slower than memory.

You still have pathological disk io even with swap disabled because Linux will evict file-backed pages. More aggressive OOM killing is currently the only solution, which is not ideal, but losing one process beats losing all of them. I use earlyoom, and I assume systemd-oomd is something similar.

> Which problem?

Thrashing and becoming unusable for a long time.

Reply on news.ycombinator.com