Modernizing swapping: virtual swap spaces
lwn.net
lwn.net
Windows remains mostly fully responsive even when memory is being pushed to the limits and swapping gigabytes per second, while on linux when I ran a stress test that ate all the memory I had trouble even terminating the script
The kernel is very slow to kill stuff. Very very very very slow. It will try and try and try to prevent having to kill anything. It will be absolutely certain it can reclaim nothing more, and it will be at an absolute crawl trying to make every little kilobyte it can free, swapping like mad to try options to free stuff.
But there are a number of daemons you can use if you want to be more proactive! Systemd now has systemd-oomd. It's pretty good! There's others, with other strategies for what to kill first, based on other indicators!
The flexibility is a feature, not a bug. What distro are you on? I'm kind of surprised it didn't ship with something on?
You'd think after 30 years of GUIs and multi-tasking, we'd have this figured out, but then again we don't even have a good GUI framework.
As far as I know, Linux still doesn't support a variable-sized swap file, but it is possible to change how aggressively it over-commits memory or kills processes to free memory.
As to why there differences are there, they're more historical than technical. My best guess is that Windows figured it out sooner, because it has always existed in an environment where multiple programs are memory hogs, whereas it wasn't common in Linux until the proliferation of web-based everything requiring hundreds of megabytes to gigabytes of memory for each process running in a Chrome tab or Electron instance, even if it's something as simple as a news article or chat client.
Check out this series of blog posts. for more information on Linux memory management: https://dev.to/fritshooglandyugabyte/series/16577
You can add (and remove) additional swapfiles during runtime, or rather on demand. I'm just unaware of any mechanism doing that automagically, though.
Could probably done in eBPF and some shell scripts, I guess?
I once was trying to set up a VPN that needed to adjust the TTL to keep its presence transparent, only to discover that I'd have to recompile the kernel to do so. How did packet filtering end up privileged, let alone running inside the kernel?
I recently started using SSHFS, which I can run as an unprivileged user, and suspending with a drive mounted reliably crashes the entire system. Back on the topic of swap space, any user that's in a cgroup, which is rarely the case, can also crash the system by allocating a bunch of RAM.
Linux is one of the most advanced operating systems in existence, with new capabilities regularly being added in, but it feels like it's skipped over several basics.
But Linux also overcommits mmap-anonymous/sbrk, while Windows leaves the decision to the user space, which is significantly slower.
pipe (fd[2]); // create a pipe to share with the child
if (fork () == 0) { // child
close (...); // close some stuff
setrlimit (...); // add a ulimit to the child
sigaction (...); // change signal masks
// also: clean the environment, set cgroups
execvp (...); // run the child
}
Comparing it to Windows is hilarious because Linux can create processes vastly more efficiently and quickly than Windows.
In 1970 it might have been the only way to provide a flexible API, but nowadays we have a great variety of extensible serialization formats better than "struct".
Actually, fork(2) was very inefficient in the 1970's and for another decade, but that changed with BSD 4.3 which shipped an entirely new VMM in 1990 in 4.3-Reno BSD, which – subsequently – allowed a CoW fork(2) to come into existence in 4.4 BSD in 1993.
Two changes sped fork (2) up dramatically, but before then it entailed copying not just process' structs but also the entire memory space upon a fork.
Or, to quote a paper on deficiencies of fork, "fork() tremendously simplifies the task of writing a shell. But most programs are not shells".
Next. A first solution is trivial: make (almost) all syscalls to accept the target process's pidfd as an argument (and introduce a new syscall to create an empty process in suspended state) — which Windows almost (but not quite) can do already. A second solution would be to push all the insides of the "if (fork () == 0) { ... }" into a eBPF program and pass that to fork() — that will also tremendously cut on the syscall costs of setting up the new process's state as opposed to Windows (which has posix_spawn()-like API).
> create duplicate worker processes.
We have threads for this. Of course, Linux (and POSIX) threads are quite a sad sight, especially with all the unavoidable signalling nonsense and O_CLOFORK/O_CLOEXEC shenanigans.
Nothing to do with overcommit either. Why would that make a difference either? We're talking about interactivity under load. How we got to the loaded state doesn't matter.
Of course the browser is the largest process in my system, so when I notice that memory is running low I restart it and I gain some 15 GB.
Basically I am the memory manager of my system and I've been able to run my 32 GB Linux laptop with no swap since 2014. I read that a system with no swap is suboptimal but the only tradeoff I notice is that manual OOM vs less writes on my SSD. I'm happy with it.
That second pillar is actually worse for interactivity than swapping the working set, which is why disabling swap entirely isn't considered optimal.
By far the best approach is just to have an absurd amount of RAM - which of course is a much less accessible option now than it was a year ago.
https://www.freedesktop.org/software/systemd/man/latest/syst...
There's a variety of oom daemons. bustd is very lightweight & new. earlyoom has been around a long time, and has an --avoid flag. https://github.com/rfjakob/earlyoom?tab=readme-ov-file#prefe...
Your concerns are very addressable.
Honestly on the desktop I'd rather a popup that allowed me to select which app to kill. But the kernel doesn't seem to even be able to prioritize the display server memory.
1. Use `choom` to give your Firefox PIDs a score of +1000, so they always get reaped first
2. Use systemd to create a Control Group to limit firefox and reap it first (https://dev.to/msugakov/taking-firefox-memory-usage-under-co...)
3. Enable vm.oom_kill_allocating_task to kill the task that asked for too much memory
4. Nuclear option: change how all overcommiting works (https://www.kernel.org/doc/html/v5.1/vm/overcommit-accountin...)
4. disabling overcommit is another footgun, it makes malloc fail long before the memory is exhausted. See for a detailed explanation https://unix.stackexchange.com/a/797888/1027
In my experience this is only on later versions of the NT Kernel and only on NVME (mostly the latter I think).
Linux does not page out code pages into the swap. You might be conflating page reclamation with swapping instead.
In Linux, executable «.text» pages are mapped[0] as file-backed pages, not anonymous memory, so when the kernel needs to reclaim RAM it normally drops those pages and reloads them from the executable file on the next page fault once they are accessed again (i.e. on demand) rather than writing them to swap.
In this particular regard, Linux is no different from any other modern UNIX[1] kernel (*BSD, Solaris, AIX and may others).
[0] Via mmap(2) in argv[0], essentially.
[1] Modern UNIX is mid-1990's and onwards.
The problem problem though is all the times when Windows is totally unusable even though it's doing exactly jack shit. An example would be when it's doing all its pointless updates/upgrades.
I don't know what people are smoking in this world when they somehow believe that Windows 11 is an acceptable user experience and a better OS than Linux.
Somehow though it's Linux that's powering billions if not tens of billions of devices worldwide and only about 12% of all new devices sold worldwide are running that piece of turd that Windows is.
The hangs usually happened when I was stressing VFS (the computer was a samba server) along with other workloads. To trigger a hang manually I would read in large files (bigger than available ram) in parallel while running a game. I could get it to hang even with 128GB ram. I tweaked all the vfs settings (swappiness, etc...) to no avail. I tried with and without swap.
In the end it looked like memory was not getting reclaimed fast enough, like linux would wait too long to start reclaiming memory and some critical process would get stuck waiting for some memory. The system would hang for minutes or hours at a time only making the tiniest of progress between reclaims.
If I caught the problem early enough (just as everything started stuttering) I could trigger a reclaim manually by writing to '/sys/fs/cgroup/memory.reclaim' and the system would recover. I wonder if it was specific to btrfs or some specific workload pattern but I was never able to figure it out.
One day I will probably see if I can still reproduce the original problem and be more methodical about it. More likely on list of things I might not ever get around to.
On GNU/Linux and the rest not supporting dynamic swap files, you can swap into anything ensembling a file, even into virtual disk images.
Also set up ZRAM as soon as possible. 1/3 of the physical RAM for ZRAM it's perfect, it will almost double your effective RAM size with ease.
It worked very well with my preceding laptop limited to 4GB of RAM.
In a sane world the user would decide which application to shut down, not the OS; the user would click the appropriate application's close gadget, and the user interface would remain responsive enough for that to happen in a matter of seconds rather than minutes.
I understand the many reasons why that's not possible, but it's a huge failing of Linux as a desktop OS, and OOM killers are picking around the edges of the problem, not addressing it head-on.
(Which isn't to say, of course, that OOM killers aren't the right approach in a server context.)
Isn't the only reason that the UI effectively freeze under high memory pressure? If the processing involved in the core parts of the UI could be prioritized there's no reason for it not to work?
I don't understand why we can't have (?) a way of saying "the pages of these few processes are holy"? (yes, it would still be slow allocating new pages, but it should be possible to write a small core part such that it doesn't need to constantly allocate memory?)
Not a bad system, but macOS has a fundamental memory leak for the past few versions which causes even simple apps like Preview.app, your favorite browser (doesn't matter which one), etc. to 'leak' and bring up the choose-your-own-OOM-kill-target dialog.