Systemd 248 RC3: systemd-oomd is now considered fully supported
github.com
github.com
It's a heck of a problem to deal with, and a OOM killer will go part of the way to fixing it, but I still have two complaints about the thrash problem.
1. In similar situations on other operating systems, I don't usually see the GUI freeze to an unusable state. In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. I suppose it's possible that I just haven't used other OSes enough in a long time and they have this too, but this seems like a solvable problem. I assume it's not a CPU issue, since the scheduler should handle that just fine. So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?
2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it (even if the OOM killer blames something else). Now, I understand that in theory unused memory is wasted memory. So a browser using a ton of memory can be a good thing. But this is only true when memory the browser is holding but not using can be reclaimed to be used by another program, and this seems to basically never happen. If I close and reopen the browser (with the tabs automatically restored) the memory usage drops by 80% or more; the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I suspect solving these two issues would largely fix thrashing for most desktop users, without the need to ever kill anything, which is obviously undesirable.
Feel free to tell me if one or both of these is incorrect or impossible. These are just my observations, I'm not a kernel dev or anything close to one.
You can recover system from this state by manually triggering OOM killer using Magic SysRq keys. (Alt+SysRq+F, provided you have SysRq enabled in /proc/sys/kernel/sysrq)
Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up.
If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen:
* Memory rises
* Some pages get swapped out
* Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes
* Memory rises some more
* You see more pages get swapped out
* (...and see more disk writes)
* Memory rises some more
* You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process.
* Disk queue lengths rise
* One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up)
This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk.
But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit.
So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out?
NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.
I suppose the solution is to figure out how to reduce or eliminate the number of disk reads after the DE initially loads, or cache everything needed in memory with a "do not evict" flag on it. Especially for the TTY, that's basically a crucial system function and keeping everything needed for it always available in memory wouldn't take much space relative to a typical desktop system.
I think at least some of the problem is the program code pages getting evicted to disk, so they have to be read back to memory too, which goes to the back of the disk queue. When I'm experiencing thrashing, even basic stuff like alt-tab to switch windows doesn't work. And I assume the window manager doesn't need to do disk reads to switch windows. Even the damn X cursor freezes!
I was able to get glitch-free audio by tweaking software and soft-irq realtime and non-realtime priorities, when the default Ububtu (with low-latency kernel) was nearly unusable for audio.
Which problem? Yes, there's always disk contention. But if you don't have swap enabled then you won't have pathological swap activity dominating the disk.
There are several bad things that happen when you run out of memory. One of them is pathological swapping. Another is loss of disk caches. Another is (eventual) oom. The one that has the most detrimental effect, and typically results in the system having to be reset, is swap activity saturating disk io. Even when you've got fast SSDs, because they're still orders of magnitude slower than memory.
Thrashing and becoming unusable for a long time.
This! Always wanted a /proc/$PID flag to flag process' pages never to go to swap and reserve a number of cache pages specifically for the process. Something similar to mlock.
Perhaps this falls outside the category of "any memory condition"... :)
Though, it was responsive. :D
Needless to say, I think 1 is harder than expected. Fedora has made some recent changes on the OOM killer front though, perhaps those are closer to what you're interested in.
* I run separate cgroups for admin SSH with (1G min, 4G max ) - worst case scenario I can ssh into the desktop and kill offending processes.
* Separate cgroup for two browsers ( 8G each max ).
* Separate cgroup per qemu VM ( VM max ram + 512M each max ).
After implementing that I went from periodic lock ups to a quick browser crash and a restart maybe once a week.
The "Auto Tab Discard" extension[1] for Firefox unloads tab contents, keeping only the title and favicon. Tabs can be put in an allowlist to prevent discarding. I don't know if Chrome or Safari have something similar.
[1] https://addons.mozilla.org/en-US/firefox/addon/auto-tab-disc...
> Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I went the ballistic route: most of my browsers run inside local Docker containers, on which I put CPU and RAM quota.
It's my machine: I decide how "citizens" behave : )
#!/bin/sh
systemd-run \
--user \
--scope \
--property=MemoryHigh=4G \
--slice=background \
/usr/bin/firefox "$@"
MemoryHigh=bytes
Specify the throttling limit on memory usage of the executed processes in this unit. Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases. This is the main mechanism to control memory usage of a unit.
MemoryMax=bytes
Specify the absolute limit on memory usage of the executed processes in this unit. If memory usage cannot be contained under the limit, out-of-memory killer is invoked inside the unit. It is recommended to use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense.
You can put resource limits on basically anything (see systemd.resource-control(5)).
In Linux I was never been able to generate a similar behaviour. No matter how aggressive I set the swappiness, the system didn't began to swap until it was practically locked up with the mouse pointer lagging.
Windows reserves some memory for its functions, for example the UI (maybe the fact that the GUI is in the kernel helps), and thus doesn't have these kind of problem, especially with an SSD (and I know that page file on SSDs is not ideal... but had a disk for 5 years swapping on it and never had a problem).
The mouse cursor did move smoothly throughout, though, as I recall.
Oh, what fresh hell is this?
Honestly, it seems great and not very tricky for a system which already manages namespaces with private mounts.
Ghosting: by bind-mounting over the system fs -you might introduce a bug/condition that doesn't show up in a recovery boot scenario, or forensic analyzis of the filesystem/disk - you could introduce "ghost" files/artefacts.
Yes, sure. That doesn't stop hundreds of vendors with custom install scripts who will use /usr and /opt which is the current state of things. Systemd actually provides a workaround here which stops that package from changing those hierarchies globally, which... seems like something you'd want?
> you might introduce a bug/condition that doesn't show up in a recovery boot scenario
Why would you try to run anything but recovery tools in a recovery boot scenario?
> or forensic analyzis of the filesystem/disk
You can reproduce the same environment using the same overlay configuration - that setup is not hidden from you. It's not any more confusing in forensic than a docker service and we deal with them (including bind-mounts) just fine.
Mixing this with OS package management will require some consideration.
So they are using overlayfs, which means that "upper" directories have visibility precedence on file name conflict. Which means that some auditing facility should be required that alerts if an upper sys image masks a lower one, or if an OS package some time later installs a file to the "lower" directory, perhaps the original rootfs, but a sysfs package still takes precedence…
Tying them to pid 1, and forcing them to be used all the time for some hacked up thing where /usr is different depending on which process is looking at it? Not so much.
It’s no more privilege than pid 1 had before because of .mount units.
You know this doesn't do anything unless explicitly configured in the system and/or unit configuration?
https://media.ccc.de/v/arch-conf-online-2020-6390-linux-memo...
Q&A Questions: https://gitlab.archlinux.org/archlinux/conf-files/-/blob/mas...
I started reading about systemd-oomd, then followed the link to its manpage (at https://www.man7.org/linux/man-pages/man8/systemd-oomd.8.htm...) then followed a link from there to this page: "In defence of swap: common misconceptions" at https://chrisdown.name/2018/01/02/in-defence-of-swap.html).
The first line of the tl;dr is: "Having swap is a reasonably important part of a well functioning system. Without it, sane memory management becomes harder to achieve."
Now, afaik linux as a kernel is pretty much designed to have swap memory and really not meant to be ran without it. Except disabling swap is required to run kubernetes, so much so that the kubelet will plain refuse to start if it detects swap memory enabled.
Why?
Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process.
It used to be that the recommendation for swap was 2x ram. This recommendation was from the days were single-digit mega bytes of ram was common, and also from when linux core-dumps were both expected, and stored to swap.
Fast forward to 2010ish. You might have a 64GB or 128GB SSD, but you probably also have a spinning drive. you have 8 or 16GB of ram, so your 16-32GB swap would be 1/4 the SSD, plus you heard that putting swap on an SSD wears it out faster, so you put a 16-32GB swap file on the spinning drive.
Lets say you are developing software and you accidentally allocate in a tight loop. At some point the majority of the pages in ram not from the malloc loop end up in swap, so you try to do something to kill it, but Xorg and xterm both have their pages scattered across spinning drives, and your program is allocating memory faster than your machine saps in. If you are patient enough to not hard reset, an hour later, swap fills up and the OOM killer starts killing programs, and probably (but not definitely) kills the buggy malloc loop before it kills something more vital.
You are annoyed, so disable swap and do the same thing: the OOM killer kills probably just your web browser (which is virtually indistinguishable to a malloc loop with Web 2.0), and the malloc loop. The system recovers in minutes. You now swear off ever running with swap again.
Today: If you have a spinning disk 1GB or swap on it is fine, but zram is another good alternative. If you have slow and/or fragile SSD storage (e.g. sdcard or eMMC), zram is a good option. If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.
Not sure about Linux, but under macOS I had a bug that was leaking uncompressed 12 MP photos in a loop. I was browsing the web while waiting for my process to run and I got up to something like 150 GB of swap used (macOS uses dynamically growing swap files) before I even noticed that there was something going on.
It always boiled down to never allocate more swap than you're willing to wait, and memory capacity multiplied over the years far faster than real world drive thruput.
So I was willing to slow everything down to use 8 megs of swap in 1994 in order to have enough virtual memory to compile a kernel, and in 2021 I'm still willing to slow everything down to use 8 megs of swap, in fact drive thruput has maybe quintupled since then, maybe, so I'd be willing to slow my system down into using 32 megs of swap in 2021.
The problem is 8 megs of swap in 1994 dramatically increased my horizons and I could use it to compile the kernel and stuff like that. In 2021, 32 megs of swap just makes my web browser OOM kill about 100 milliseconds later so why bother?
If the kernel sees that some memory is barely, if ever used, it will swap it out preemptively. This gives you more memory available for both programs and temporary caches.
As such it’s still advantageous to add much more swap than this, since it effectively gives you extra free space in RAM at essentially no cost.
We are running a distributed in-memory olap database. All nodes have 500gb ram used basically at capacity. One node has 1.5T of nvme swap (initially just for testing but it works well) so it hosts a shard and the manager/reduce node. We've tested forcing the nodes onto swap and saw a x10 performance decrease, but our implementation is pretty shit when it comes to memory access patterns so it's not too bad. The collocated node is slightly slower in terms of performance but doesn't impact total response by a meaningful margin.
Why can you not get swap usage? It’s in smaps, right?
The historic reason Kubernetes didn't let you run with swap is that it's complicated from a resource management perspective - it's not clear whether each pod should be requesting some amount of swap the same way it requests some amount of RAM, and also the container runtimes didn't really have support for that (largely because you need cgroupv2 to have a hope of that sort of per-cgroup swap resource control working at all). But it's not an insurmountable problem, and it sounds from that ticket like they recognize that letting systems have swap enabled for the sake of a userspace oomd is an important use case.
Swap comes with a lot of caveats -- when a performance-critical process begins swapping it can on occasion be more useful to just let it OOM instead.
My personal desktop has 64GB of memory, with no swap enabled. Why would I want swap it I don't need more than that?
In my experience, even with ample swap, when Linux gets memory starved it goes haywire anyways.
I'm an ex linux sysadmin, and haven't run swap on my personal systems in years.
If you filter out the RFEs (Requests For Enhancement) you find 617 issues. If you filter for only issues labeled as a bug you get 97 results.
In my experience, unless you can get Red Hat to care about patching a specific bug, you can rely on opened bugs to stay open.
> don't always label bugs that way (example: systemd/#5616)
Nobody figured out if that issue is in logind or X yet. They did not close it so it's not like they say it can't possibly be on systemd side. (typical Lennart comments aside)
Finally the author of nohang is working on arguably superior solutions: https://github.com/hakavlad/prelockd https://github.com/hakavlad/memavaild https://github.com/hakavlad/le9-patch It's unfortunate that systemd has chosen to integrate an inferior solution
It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd https://github.com/facebookincubator/oomd , whose documentation gives a bunch of reasons as to why you would prefer a userspace oomd that takes in PSI data and can be configured to proactively kill misbehaving processes instead of just letting the kernel OOM killer handle it. The major reason is time to recovery: a misbehaving process can cause a system to be so far under pressure that the kernel OOM killer will take a long time to flush things out, but a userspace component can respond in advance with more configurable rules (and more flexibility, since the kernel doesn't believe you're at capacity yet).
> It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd
Yep, I authored the original standalone oomd (along with many others' help) and Anita has been working on systemd-oomd. There's plans to converge and simplify everything that's been learned from oomd into systemd-oomd so the community can benefit from all our experiments.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`. I can easily see services dependencies, I can easily change the user running the service, the workdir, etc. There is a standard interface to start, stop, restart, enable at startup. All of this from one system that is consistent and the same between Linux systems.
But really the best features IMHO is journald.
My perspective is that on the one hand, systemd is very 'cerimonial' in how it goes about things, while shell-based systems have the potential to be extremely well done. My go-to example is the OpenBSD init script - I once tried to boot OBSD over the network, and mount it on a RAM filesytem, and was able to simply open that boot script and change the mount commands. Ever since, I've been impressed by the clarity and simplicity of that shell script (in its original form, not after I was done with it (: ).
> Because sh and bash are horror shows.
I disagree. The syntax is obtuse, but it is at this point very well understood, and the rest of the system runs on it. Better to use fewer languages throughout, than more. Are you referring to something other than the syntax?
> Maintaining and creating new service init scripts is a real pain and difficult to do with consistency.
I'm not sure, but it sounds like you're talking about systemd services. If you are, I agree; multiple files with layers of cerimony are hard to compare to an extra line of shell core in an rc.local. If you're talking about non-systemd, I've never encountered this, and certainly found the process far less painful than the systemd way.
> Bash scripts can do anything, meaning they have to be audited one by one, every single time they are updated.
Just diff the scripts...not that hard. Just have to check the changes. That's no harder than any other program you care about.
> Every single distribution has its own way to run services via scripts.
vive la différence! No point to the stagnation caused by homogeniety; I don't see why I should expect debian services to run the same as FreeBSD.
> And that can change between upgrades.
Systemd is notorious for changing things. One great example is when they tried to change how processes can persist after the user logs out. I don't think you have a leg to stand on here.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`.
I can too: `cat /etc/rc.local` and `vim /etc/rc.local`. Added benefit, I'm using the same tools and interfaces that I do for the rest of my system administration.
> I can easily see services dependencies, I can easily change the user running the service, the workdir, etc.
Service dependencies are the strong point of systemd, I will give you that. However, `su` can change the user running the command, and `cd` is this awesome little tool I found for changing the directory I'm in. No extra benefit there, just new commands for doing the same old things.
> There is a standard interface to start, stop, restart, enable at startup.
That standard interface is only useful because the underlying complexity has increased so much that `systemctl start` has become a necessary abstraction to cover it up. Your standard interface is medicine for an ailment I do not have.
> All of this from one system that is consistent and the same between Linux systems.
Why is that beneficial? It just means that you've put all the innovation in the hands of the systemd devs. I'm over on systems like void, where my startup times are faster, my base system load is a third, and I can get my hands dirty in the system internals without googling for strange systemd incantations.
> But really the best features IMHO is journald.
You're going to have to spell that out, because I really like being able to use a proliferation of text-oriented tools to manage my logs. I'm not a fan of binary, no matter the supposed speed improvements.
Well, there must be room for such organizations too. It's a colorful world after all.
Systemd is an umbrella under which many key userspace tools are being developed. The actual init system itself is a lot bigger than the old SysV Init system was, but it is separate than all these other things that also say Systemd on them.
The best analogy is GNU. You don't complain that GNU contains a compiler, and also a text editor, and also C System Library do you? GNU is a project that develops a load of components that play well with each other. Systemd is the same.
The increase in the amount of user-level software that depends on systemd and Wayland is a forcing factor. For example, my understanding is that official distributions of GNOME depend on systemd, and many Linux desktops are being actively developed for Wayland (though I don't think any major desktop environment or toolkit has abandoned X11). I expect this trend to continue, given that systemd is part of the most popular Linux distributions, Wayland will be default in Ubuntu (https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-2...), and GNOME runs on Wayland (with KDE's port to Wayland being actively developed: see https://www.phoronix.com/scan.php?page=news_item&px=KDE-2021...).
So, yes, I do see a split occurring between "modern Linux" (with systemd and Wayland) and "traditional Linux," with the BSDs fitting in closer to the "traditional Linux" camp.
There will be Linux for desktops to run a web browser (aka poorly emulate a chromebook) and for docker and docker only.
If you want a non-containerized server or some kind of non-desktop embedded thingy, or anything that can be computed other than emulating a chromebook and doing containers, you'll have to run FreeBSD or some other OS.
The days of "just install Debian to do anything anywhere" are gone, which is sad, because there's really no technical reason or need for the destruction.
What are you missing from Debian these days that was there 5/10/20 years ago? What has been "destroyed"?
Including busybox in the initrd or whatever it's called now is probably a good idea. Having systemd developers write a new interpretation of that, rehashing old bugs is not great.
which uses LLVM for building Linux and Systemd, replaces all core utils with non GNU versions. It use a non GNU libc and so on...
But that aside people often confuse/mixup the systemd service manager and the systemd project which bundles a wide variety of programs which can be used with systemd.
The reasons for this is because systemd itself confuses this. The change log is not split by sub-projects and the sub-projects have sometimes no clean boundary between them and the systemd service manager.
I think it would be nice if systemd changes in three ways:
1. Have more clean boundaries between the core service manager and it's sub-project.
2. Have a recurring periods where new features are blocked from being merged and development focused on cleaning up (e.g. fixing bugs). (I can recommend that to any projects, very few do so).
3. Get "outside" feedback for certain mainly UX aspects, some parts of systemd are out right counter intuitive unless you are not very familiar with it. A thinks which is very easy to happen to any project, but still could be improved on (and maybe already is improved on).
Oh and there where a very few security WTF are you doing moments, I hope there will be no more such moments in the future as systemd on itself it quite useful for Linux, especially desktop Linux.
Just wait until ls has to have hooks into dbus to do its job.
Oomd, oh yea, good idea, but WHY it has to be in systemd? The same applies to about 3/4 of things in there. Because of this i've started to call it LennartOS. And also I don't feel like linux is about freedom anymore but more like some sort of weird power display games.
I use several of these projects because I like the systemd design philosophy (ex the way the CLI interface and config files are) and trust the developers of their software. However, there are popular altenatives available for all of these components, should you choose to use them:
systemd-resolved: bind9, dnsmasq, powerdns
systemd-networkd: NetworkManager, ConnMan
systemd-boot: GRUB2, Refind, EFISTUB
systemd-oomd: earlyoom, nohang, oomd (the facebook one, not systemd)
It's a service manager for your Linux OS which manage all services down to initial startup related services.
Just viewing it as a init system is misleading.
As a side note I think a service manager directly above the kernel is a must have for any modern server or desktop operating system ;=)
Systemd is the name of the init system. Systemd is also the name of the project developing multiple systemd-* services. Maybe they could help the situation a bit by renaming the service itself systemd-init, but we've had enough time for people to learn to deal with a tiny bit of ambiguity...
However, you should probably switch to FreeBSD because you think it’s great too, not because you hate systemd.
«It (systemd) provides a system and service manager» (from https://www.freedesktop.org/wiki/Software/systemd/).
So it manages services and manages the system, as a whole.
> Every time I see all the stuff packed into it I feel extreme urge to switch to freebsd again.
Meh, a lot of people bring this up, but they never switch.
Try understanding a bit more than just the name's prefix before getting so bent out of shape.
Lennax
https://papers.freebsd.org/2018/bsdcan/rice-the_tragedy_of_s...
Joke's on you.