Earlyoom – Early OOM Daemon for Linux | Hacker News Reader

Earlyoom – Early OOM Daemon for Linux

github.com

50 pointsharporoeder48 comments

Does this handle cgroups such that it would know to kill chrome even though each individual tab is only using a little bit? Could be mistaken but looking for the single largest process can be misleading if you’re dealing with applications that have forked a ton of processes…

systemd-oomd is another thing that exists, and does of course take cgroups into account.

But since chrome should survive a tab dying, just killing an individual tab should be a perfectly good strategy in most cases.

The problem is not with killing a single tab rather all of chrome, the problem is with identifying that chrome is the memory hog in the first place which you won't see if you just look at individual processess because the memory usage is spread out.

What is the use case where you'd want to kill all of Chrome just because there are many tabs open? If there is a memory leak in Chrome, it will almost certainly affect a single tab process, and that can be killed. If you just need a bit more memory, killing one of Chrome's tabs may be a good choice.

Otherwise, it's very likely that Chrome is more important than other processes running on the system, in any use case where a heavily used Chrome is present, isn't it? You won't have Chrome with a thousand tabs on a server system, and if you're running something like a massive compilation, you probably just want it to fail if it can't fit in RAM, not kill the whole Chrome process group to continue.

> What is the use case where you'd want to kill all of Chrome just because there are many tabs open? If there is a memory leak in Chrome, it will almost certainly affect a single tab process, and that can be killed. If you just need a bit more memory, killing one of Chrome's tabs may be a good choice.

They aren't saying they want to kill all of chrome. They're saying that in order to know that the problem is that chrome is using all of the memory in order to decide to kill a single chrome tab, the daemon needs to be able to identify that all of the processes belong to one program (chrome).

I don't think I understand. Isn't it enough that each tab is treated as a separate process, so the normal per-process OOM logic can decide to kill a tab process just like it might decide to kill, say, Thunderbird?

Let’s say you have some photo editor using 3 GiB of memory and Chrome using 20 GiB of memory because a lot of tabs are open. Any given tab is < 1 GiB. What should you kill? Logically, you’d want to start killing Chrome tabs as they’re your best bet to recoup the most amount of memory even though no single process is the largest but “kill largest process” is going to kill the 1 GiB.

Another instance of this is a shell fork bomb - your memory is used up but each individual fork uses very little and killing the largest process won’t help.

I have taken to running this anytime I start a jupyter notebook using more than a trivial amount of data. It has only fired off a couple of times, but I have been grateful for the safety net.

Anytime I can consume my 64G of ram, I was probably doing something stupid anyway.

Advent of code? :)

I use it on a 4GB PI in combination with XSuspender. It has its drawbacks, but works surprisingly well.

XSuspender - Automatically suspend inactive X11 applications When an application window loses focus, XSuspender tries to match it to one of the rules in its configuration. If a match is found, the application is sent a SIGSTOP signal (preventing the process from obtaining further CPU time). Upon windows regaining focus, the process is seamlessly continued where it had left off.

Wouldn’t this suck for power on laptops? Doing something at a fixed frequency (especially as high as 10Hz) will cause CPU wake ups where previously you wouldn’t need any. Say goodbye to your battery life.

For an awake power state you almost certainly have at least one core waking up more frequently than 10Hz just to handle poking devices like the GPU. HW power management (such as through wait for interrupt instructions) operate on a much smaller time scale and so you wouldn't really be perturbing anything on a real interactive system.

On phones in android we even aggregated “once per hour” wake ups to save power noticeably, so it does matter actually.

Wake from suspend-to-RAM (and likely with a radio power up) is going to be gobs more expensive than a wake from retention because you're both coming from a much lower power state (you were doing great before and now you're not, so a big change in energy usage over baseline) and it has a high overhead (the core cluster/SoC has to go through power on reset and get reconfigured by SW). This greatly contrasts with a wake from WFI where at minimum all architectural state is retained but generally caches and predictors are retained for quite a while.

I think you might be talking about different wake ups.

Why this and not Meta's oomd?

Or systemd's systemd-oomd https://github.com/systemd/systemd/pull/15206

The answer for both of those is "if you're unfortunate enough to be running a 2.6 kernel linux server from 2005, then you can't use cgroupsv2 and thus can't use oomd or systemd-oomd".

That is the only thing earlyoom is good for at this point I think, is old systems without cgroupsv2

Actually no there are other reasons. Meta oomd requires you to have swap enabled, which makes it a non-starter for us. systemd-oomd has its own set of drawbacks. We evaluated all options at length and earlyoom came out as the clear winner for our usecase.

Why weren't you using swap?

For our usecase (Ceph), if one server starts swapping, it completely kills the entire cluster performance. Killing the service (with earlyoom) is preferable in most circumstances.

I'm not the parent. But in the last 5 or 8 years, Linux swap has mostly been useless to me. I run a desktop with 16GB ram, have swappiness set to 0, and even when using maybe 5GB or 7GB for running programs, it wants to swap out all kinds of stuff. When I switch to another running program I have to wait for it to get taken out of swap. But I was using that running program just a while ago, why do I have to wait?

And it doesn't help in thrashing in Linux, it does it anyway.

In defence of swap: common misconceptions[0]

[0] https://chrisdown.name/2018/01/02/in-defence-of-swap.html

No misconceptions here ;)

Still the question is open, what does swap do for me in 2023? I only see disadvantages.

I spent a few days trying to configure systemd-oomd to work with my machine but I kept getting hangs when running out of memory. earlyoom is a lot simpler and thus more reliable. It only took ~5 minutes to install and test to make sure it worked and I've never had a hang since then.

Or Android’s lmkd?

If you want stuff to die inexplicably, use this! (I used to, bad idea)

I would prefer a process on my desktop be killed rather than my system become unusable to the point of requiring a hard reset of the machine as is common on Linux systems during low memory scenarios.

I've never had an OOM on Linux requiring a hard reset, but you do need some patience for the kernel's OOM killer to kick in – up to a minute or two, perhaps even a bit bit longer (I never timed it exactly).

I've had it trigger regularly, it fires far too late (if at all) and the system had already become unstable at the point in which it freezes. It should be firing far earlier.

Yeah, the system locks up, but it should recover – eventually. You just need to go for a coffee, or bagel, or bowel movement, or wank, or whatever. At least, that's been my experience.

If you have too much swap configured it can take hours to recover. Maybe days, never let it go that long without giving up. The kernel is terribly slow at swapping stuff back once memory pressure is gone.

Of course the remedy is to have no or very little swap configured, but that's not obvious when compared to other operating systems (for example on Windows more swap never hurts, and the system recovers very quickly once memory pressure is gone)

That and there are some processes that should never be swapped, and yet they are. I think it should be possible for a process to request that it is never swapped, or to suggest sections of it are never swapped. Even a swap blacklist would be a great start.

No, it doesn't always. There has been a few times I've left a machine for literal days (when I really needed it to recover) and it does not recover.

My guess is that something causes a kernel panic, like the filesystem getting itself into trouble with the hardware.

I have a work server with 384GB of RAM, some swap on spinning rust, and a fairly old installation of Linux, and it has certainly had times when it has taken hours for the kernel OOM killer to sort the thing out. earlyoom was one of the first things I installed on it. I think the more RAM you have, the worse the OOM behaviour. I also think that recent versions of Linux are much better at this than they used to be.

If your kernel has magic sysrq enabled you can use Alt + SysRq (aka Print Screen key) + F to manually invoke the OOM killer so you don't need to wait.

Usually it is a process gets killed, its children keep running uselessly, and it's so conservative that most times it was going to work fine.

In fact it has been completely fine for me to remove it.

The time I wasted having to kill orphaned processes and restarting everything, daily, was easily greater than the hard resets, which happen like once a year or so to me.

Turning off swap should give you this result. Then instead of trying to move stuff to swap which causes the system to be slow or unresponsive, the kernel will just start killing processes.

No, sadly. This doesn't happen in my experience. The main reason seems to be that executable code is memory mapped from disk anyway. So in a low-memory scenario, that gets evicted, causing the same kind of thrashing but worse because it's essentially forced to be focused on really active areas instead of the least active. The end result is instead of a slow bogging down like you get with swap enabled, you get a sudden freeze which may eventually resolve itself but during which the system is almost completely unresponsive. In general if you are running without swap I would strongly recommend you do use something like earlyoom.

Agreed - most people don't realise, but even if you don't have any swap, all your executable code is effectively treated exactly the same way as swap (except it's read-only and needed more often).

I'd love a setting on Linux to tell it to treat executable pages memory-mapped from disc as Really Important and never evict them. It'd have to come with a suitable warning that if you set this then you get exactly what you deserve, but TBH it probably is what most people actually think is happening already and want.

Heh, thanks for explaining this. Maybe my general aversion towards swap wasn't fully based on facts.

Not guaranteed, there's more than swap that the kernel can use, such as mmap.

Disabling swap is a very coarse optimization. The ultimate problem is performance. If you have 16GB RAM + 10GB of stuff sitting completely idle, that works perfectly fine. If you have 16GB RAM + no swap + a process that requires 15GB RAM and uses all of that constantly, it's going to be painful, as the kernel tries to compensate by squeezing memory out of anywhere it can, over and over.

Does it has to be inexplicably? Is there any reason the OOM Daemon can't nicely prompt you about what process they just killed? Or even ask you if you want to kill, similar to how Firefox will ask you if you want to kill a misbehaving tab?

I guess they could attempt to do so, but under extremely severe low memory conditions, the prompt (especially if its graphical) has a good chance of not reaching the user's screen in any reasonable time. That's why it has to decide without external I/O.

You can suspend a process, drop some caches, and even swap them to disk, if the memory approaches critical levels, or better yet, you can set a watermark that suspend the process and prompts you before it gets to that.

It just picks a big one at random. Like the shell in which you're running make, and then you have a mess.

Even worse when it kills containers.

Came here to tell about zram - swap in compressed RAM. Disable (or never set up in first place) on-disk swap and use compressed RAM as swap. This gives you swap, so you have advantages of that. But when you actually run out of memory, kernel oom has a chance to do it's thing, because system is still responsive in this situation.

Sincere doubt: is it really needed after MGLRU?

Yes. MGLRU only keeps the system responsive enough so that the OOM daemon can do its job when it comes to it.

Reply on news.ycombinator.com