Hacker News Reader

Comment by 0xcde4c3db | Hacker News Reader

> Warning signs of a genuine low memory situation that you may want to look into:

> available memory (or "free + buffers/cache") is close to zero

> swap used increases or fluctuates

> dmesg | grep oom-killer shows the OutOfMemory-killer at work

How about this one: the entire system becomes so unresponsive that even a VT switch takes multiple minutes to process, never mind actually logging in and running diagnostics. I don't know exactly why, but I used to get this pretty regularly on a 2 GB laptop if I did the wrong thing in Firefox. The workaround was installing earlyoom [1] (after which the tab would crash instead), although I assume there's some kind of sysfs parameter that I might have tweaked instead.

[1] https://github.com/rfjakob/earlyoom

View on HN

The reason this happens is because due to memory pressure, the IO cache is emptied and thus every IO request (even if reading just one byte) is passed through to the actual storage device. This quickly fills up the device’s queue and IO performance can no longer keep up with the system’s demands even for normally-benign stuff such as writing logs.

How about when you can't ssh to the server because ... oom-killer killed sshd?

Thankfully that system wasn't mine so I just got a laugh of this situation.

Still, 20 years later I still can't wrap my head around why Linux still needs oom-killer when Windows can run just fine without such functionality.

> why Linux still needs oom-killer when Windows can run just fine without such functionality

My Windows 10 installation just BSODs when Firefox fills up the RAM if the page file is disabled.

I'd much prefer the OOM killer.

why in the blue blazes would you disable the page file?

seriously asking. if you're fiddling with that in the 21st century you are Doing It Wrong.

You're missing the main point of my comment while falling to the swap trap:

Processes may crash, but kernel shouldn't, period.

If it crashes today with no swap while running 5 programs, it'll crash tomorrow with swap enabled while running 10 programs.

a kernel panic is sometimes the right thing. "you are doing a bad thing; I quit" is a valid response in extreme circumstances. turning off the page file in Windows in 2021 is an extreme circumstance.

a requirement of the kernel is that there is virtual memory space. the page file is part of what enables that on Windows.

on Win 8.0 (I think) and before, the page file was nowhere nearly as vital, and you could turn it off in certain situations. an educated user could use their knowledge to know when it was ok to turn it off.

new versions of Windows change many things; they are not simple reskins of previous windows versions. the kernel and low level stuff changes a lot, version to version. knowledge that was valid in Windows XP is very likely no longer valid on Windows 10 or 11, and certainly is no longer valid in all situations.

keep your knowledge up to date. even things that appear to change relatively slowly, such as Windows, experience a lot of churn at lower levels. the APIs may not change, but their implementations often do.

>My Windows 10 installation just BSODs when Firefox fills up the RAM if the page file is disabled.

I never had that happen. What actually happens is firefox just crashes, or the screen goes blank for a few seconds while the graphics driver resets.

I get all of those, too. BSOD occurs less often when the system was already pressured with several cycles of the above.

But even the graphics driver crashing is already bad enough. If there's not enough memory, offending programs should be crashing, not drivers.

(and yes, ram sticks are good and verifed with memtest86 and testmem5)

Running with the page file disabled is bad. The system runs out of address space, hence the crashes.

> if the page file is disabled

Ouch. Never run Windows without the swap. It can be as small as the system allows (AFAIR 16Mb for Win7), but it should be enabled.

While this is the obvious footgun the problem is nobody ever tests with the swap disabled.

I've seen very weird behaviour without the swap, the most funny one was when some 3D game (sadly don't remember the name) internal timer went completely nuts without the swap and game was working just fine with the swap enabled.

That's a good point, and I was actually running with 16MB page file at that time (and my reasoning for that was exactly like yours). So unfortunately it's nowhere enough.

Funny thing is I clearly remember Windows 7/8.1 showing a warning about memory running low and then killing Firefox. On Windows 10 they managed to bork that, too.

Chrome just crashes, it seems like the fault may lie with Firefox.

Windows uses a dynamic page file for swapping so it will just keep increasing the page file size until it hits the limits of the OS or config settings. You can do the same thing in OSs like Linux and BSD but most distros still default to fixed swap partitions. For example macOS uses page/swap files instead of a swap partition by default. It's just a design choice. Both have advantages and disadvantages. Linux and BSD distros usually are more aggressive about kicking out applications in favor of keeping the system stable but the trade off with OSs like Windows is misbehaving applications can more easily take down the OS.

Windows for me frequenctly needs a pagefile 8x the size of my memory in order to service the committed memory, even though it's never touched in practice.

Hint: disable or minimize overcommit and you're good to go.

I dearly wish overcommit could be disabled per-cgroup, but AFAIK this is not possible.

malloc failing seems like a much simpler indication of memory pressure than the alternatives like PSI.

AIUI, you can always use plain old ulimit to restrict virtual memory usage per-process, which has the same outcome as disabling overcommit.

yeah, for a single process. but I’m mostly interested in doing this for a process tree (like a browser), and I don’t think this would work well in that case, since the limit would apply to each process individually.

this would work if there was a cgroup virtual address space controller, but I think those have been proposed but never merged.

In my short experience you can't disable overcommit if you're using Linux on desktop because programs using gtk/glib freak out when they can't malloc.

That works for the systems I control. Most of the time I encounter OOM shenanigans on the systems which aren't mine.

Here is where something like systemd is really nice because sshd will get restarted!

As for Windows, the big difference is no fork()

> As for Windows, the big difference is no fork()

Missing the whole picture. fork() isn't the real reason, it's that Windows guarantees memory or no memory while Linux by default overcommits (both of which isn't necessarily a bad thing - it's just different ways of managing a limited resource). NT could implement (and does, it's just not exposed in Win32 API) a non-overcommiting fork() and its memory algorithms will be the same afterwards.

The problem with `fork()` is that its usage requires over commit.

Consider the scenario of a process with 51% of memory usage, what happens when it forks? The system pretty much _has_ to allow overcommit at this point, otherwise fork() will fail.

Now, you can change the accounting to say that when the process modify a memory page (thus creating a copy), we'll check that there is enough memory _then_.

That would move the "danger" of memory being yanked from underneath you. But that means that a process that calls fork() may then run into a problem some indeterministic time later.

That is kind of a killer.

  ERRORS

    The fork() function will fail if:

    [EAGAIN]
        The system lacked the necessary resources to create another process, or the system-imposed limit on the total number of processes under execution system-wide or by a single user {CHILD_MAX} would be exceeded. 

    The fork() function may fail if:

    [ENOMEM]
        Insufficient storage space is available.

The POSIX specification is clear on this: you can hard fail. Also, as an aside malloc() can also fail and the program must be able to handle that. Unfortunately for some programmers (mainly coming from GC-aware languages), they just assume that it won't fail. (Also, most rely on fork-exec but there's posix_spawn() for de novo execution).

Again, NT does indeed have a fork function, however the Windows API doesn't expose that (and this is the hack Cygwin used when forking), but it's designed to fail instead of overcommit when all memory is exhausted.

> Unfortunately for some programmers (mainly coming from GC-aware languages), they just assume that it won't fail.

In most GC languages I've used, if malloc fails for some reason (we rarely call it ourselves directly) an error happens and, unless handled, the whole app crashes. Which is fine, BTW, because I much prefer an app that crashes and dies to one that does things with the assumption something that didn't work did.

You CAN hard fail, but not overcommitting can severely limit the system in the very common case where you fork and exec. See e.g. https://lwn.net/Articles/785430/

I've been using auto tab discard on Firefox. Its a small annoyance for users but if you are like me and have more than a few tabs open, this saves a lot of ram anxiety

I think this happens when the kernel starts swapping executable images / shared libraries in and out and keeps going for a long time because according to the kernel metrics it is making great progress, even though user space is completely stalled because running code is paged out and has to be paged back in to do anything.

earlyoom kills stuff sooner instead of trying this for a long time. I’ve also heard that some people have had some success by locking base system libraries and executables in memory.

(Please let me know if this explanation is not correct or there are other solutions worth trying)

I normally run Linux without any swap partition and this still happens without swap. Realistically is this kind of oom condition (where oom killer fails to trigger) I just end up pulling the plug.

I’ve never been able to track down the exact cause.

Code can almost always be paged out, even without a swap file, because it's memory mapped from files on disk. In fact running without swap makes this behaviour worse (i.e. it happens sooner and more suddenly) because then the kernel can't prioritise keeping currently running code in memory over keeping rarely-accessed data in memory. In general you should have some swap and run with earlyoom if you want decent behaviour in a low-memory condition on linux.

Good point.

> How about this one: the entire system becomes so unresponsive that even a VT switch takes multiple minutes to process, never mind actually logging in and running diagnostics. I don't know exactly why, but I used to get this pretty regularly on a 2 GB laptop if I did the wrong thing in Firefox.

It looks like 12309 https://web.archive.org/web/20140816132654/https://bugzilla....

Do you have swap?

It's been a while, but I think it had 2 GB of swap on a 16 GB SSD (this was one of the Linux-friendly Chromebook models, hence the small storage). In any case, reports seem to vary considerably on how this behavior varies with swap size or presence in practice.

Reply on news.ycombinator.com