Is there really no higher-priority kernel process to prevent total freeze of the system and send a SIGKILL to the culprit process when such a scenario happens?
systemd-oomd should be integrated in systemd, you can configure it to your liking and see if it improves your problem.
Chris Siebenmann discusses when the OOM killer triggers: https://utcc.utoronto.ca/~cks/space/blog/linux/OOMKillerWhen
Chris disables systemd-oomd after it obliterates his X session with no explanation: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNo...
> > [...] If the configured limits are exceeded, systemd-oomd will select a cgroup to terminate, and send SIGKILL to all processes in it. [...]
> By having the user@.service template be enrolled in systemd-oomd, Fedora made the cgroup that systemd-oomd would select to be killed be all of your processes (across all of your sessions, if you have more than one). ...
Maybe *Fedora* has fixed or improved in the last 4 years. Or maybe they don't run Fedora.
In my experience it works really well. I wonder why my computer is a bit sluggish, and find out I have several gigs in zram.
If that was in swap on a disk, it would be really painful.
Made it far easier to target any containers that got too hot rather than ever risk anything higher priority.
Just found this comment:
User-space OOM killers never really worked for me and imo are not a proper solution anyway. This option instead lets you make the kernel OOM killer actually work for desktop use.
Currently have it set to `1000` and it works very well for me (don't remember the last time I had a full system freeze due to OOM).
It's because linux is a toy OS. Specifically, it overcommits memory in the hope/assumption that it won't all be used at once, but doesn't have a way to gracefully degrade when applications collectively want to use more memory(+swap) than it actually has. You can turn off overcommit, but applications are designed with the overcommitting feature in mind, so your experience might not be as good as you were hoping for.
Making a massive swap space helps a little bit. It's better to just never let your actual memory usage go above 85% to 90%. It's fine to go above if you're trying to optimize a server with a specific set of processes to wring every last bit of efficiency out of it, but not for general desktop computing.
If it really bothers you OpenBSD (edit: thanks for the reminder TimTheTinker) and Illumos don't allow overcommit at all and Windows handles this situation much more gracefully, so WSL is an option too. If you don't mind Oracle (i do), solaris also doesn't allow overcommit.
this guide should help make it less likely for sshd or your login shell to get killed: https://ianlpaterson.com/blog/oom-lockout-ssh-survival-harde...
You should have a much better “plan A” to avoid this situation.
…by default. It can be disabled via a sysctl:
* https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
The problem with turning it off is that the system and applications have been architected assuming that it will be on, so things like fork/execing a memory heavy processes or allocating memory inside a cgroup (which still pretends overcommit is enabled and there's still no way to disable that assumption) that used to work fine might break with no good way to get them to work again. This comment (and siblings) have more specifics: https://news.ycombinator.com/item?id=27794237#27795199
In a nutshell, overcommit. It's more or less broken by design but it's also incredibly practical so pretty much everyone does it.
Couple that with the fact that it's difficult bordering on impossible to correctly determine the culprit. If you've got 16 GB RAM and the user launches 3 processes each of which attempts to use 8 GB who should you kill?
But where linux would automagically back those pages once you use them, Windows requires you to actually ask for those pages to be backed by something (physical memory or page file) by calling VirtualAlloc with MEM_COMMIT on the range you actually want to use
If that's the only issue, it's avoidable for most use cases. Lots of processes that fork are doing fork/exec to run a helper program. If they know they will do that and that they will be a large process, it's often useful to setup a fork/exec helper in early application startup.
However, there are some applications that use CoW more intentionally. Lock -> fork -> (unlock in parent / persist coherent snapshot in child) is a common pattern; I believe redis uses thst pattern and I've seen it mentioned in discussions about MMO servers. I believe postgres uses fork and CoW for transaction isolation ... but postgres also runs on Windows so there must be another way or I don't understand.
For the persist case, you could imagine some sort of flag to fork to allow overcommit and maybe even to let CoW requests stall in the parent rather than fail... the child is expected to do its work and exit in a limited time.
I'm sure there are many reasons why Linux can't do that by default, but it's a real shame.
Given that one can have swap files, and can also use LVM LVs for swap, and given that userspace OOM killers that work way better than the built in one -for some workloads- exist, I see no reason why you couldn't have this on Linux. This comment [0] mentions a project that claims to do just that -and seems to use swapfiles to do it-, but I don't have any experience with it.
FWIW, I did find the README by the original author [1] far more informative than the one written by the new maintainer.
In all cases, yes, in some cases no, you can make some heuristics for common use cases
For example, if I have 3 process hogs, on desktop I'd rather have my dev containers be killed, than anything I'm using.
Or on server, I'd rather have anything else but SSH/VPN software killed, because that's needed to debug the problem.
I recommend using the earlyoom if you want more aggresive oom kill:
https://github.com/rfjakob/earlyoom
The README contains a lot of interesting information.
It took a long evening to figure out that it gets earlyoom'd immediately because it tries to allocate too much. Previously the very familiar hitching and freezing was a very easy sign of what kind of issue I was dealing with
If you know ahead of time which programs / processes are at risk of unacceptably high memory usage, check out "ulimit".
If you are you using disk swap - not sure why would if you have a SSD, but I once heard some justification for doing that - then install early OOM.
Or zswap instead of zram.
https://chrisdown.name/2026/03/24/zswap-vs-zram-when-to-use-...
And since that happened on all nodes around the same time, it took out the entire cluster. If you are not familiar, random, uncontrolled node restarts in any kind of multi node database or search product are a great way to trigger outages. So, not great.
I've had quite a few encounters with the oomkiller killing processes that were important and didn't need killing. Or as I like to phrase it "killing the one reason this server exists".
These days the way to size a server is "have enough memory to run whatever you need running + at least half a GB for whatever apt might randomly demand at any point". And guess what, memory tends to be expensive in cloud environments so people tend to get vms with as little as half a GB of ram.
The culprit turned out to be the kernel itself, and rolling back to a 6.1 series kernel made the problem go away. I see that Linus's love for vibe-coding is already paying dividends.