Linux Ate My RAM | Hacker News Reader

Linux Ate My RAM

linuxatemyram.com

> Warning signs of a genuine low memory situation that you may want to look into:

> available memory (or "free + buffers/cache") is close to zero

> swap used increases or fluctuates

> dmesg | grep oom-killer shows the OutOfMemory-killer at work

How about this one: the entire system becomes so unresponsive that even a VT switch takes multiple minutes to process, never mind actually logging in and running diagnostics. I don't know exactly why, but I used to get this pretty regularly on a 2 GB laptop if I did the wrong thing in Firefox. The workaround was installing earlyoom [1] (after which the tab would crash instead), although I assume there's some kind of sysfs parameter that I might have tweaked instead.

[1] https://github.com/rfjakob/earlyoom

How about when you can't ssh to the server because ... oom-killer killed sshd?

Thankfully that system wasn't mine so I just got a laugh of this situation.

Still, 20 years later I still can't wrap my head around why Linux still needs oom-killer when Windows can run just fine without such functionality.

Hint: disable or minimize overcommit and you're good to go.

That works for the systems I control. Most of the time I encounter OOM shenanigans on the systems which aren't mine.

I dearly wish overcommit could be disabled per-cgroup, but AFAIK this is not possible.

malloc failing seems like a much simpler indication of memory pressure than the alternatives like PSI.

AIUI, you can always use plain old ulimit to restrict virtual memory usage per-process, which has the same outcome as disabling overcommit.

yeah, for a single process. but I’m mostly interested in doing this for a process tree (like a browser), and I don’t think this would work well in that case, since the limit would apply to each process individually.

this would work if there was a cgroup virtual address space controller, but I think those have been proposed but never merged.

In my short experience you can't disable overcommit if you're using Linux on desktop because programs using gtk/glib freak out when they can't malloc.

Here is where something like systemd is really nice because sshd will get restarted!

As for Windows, the big difference is no fork()

> As for Windows, the big difference is no fork()

Missing the whole picture. fork() isn't the real reason, it's that Windows guarantees memory or no memory while Linux by default overcommits (both of which isn't necessarily a bad thing - it's just different ways of managing a limited resource). NT could implement (and does, it's just not exposed in Win32 API) a non-overcommiting fork() and its memory algorithms will be the same afterwards.

The problem with `fork()` is that its usage requires over commit.

Consider the scenario of a process with 51% of memory usage, what happens when it forks? The system pretty much _has_ to allow overcommit at this point, otherwise fork() will fail.

Now, you can change the accounting to say that when the process modify a memory page (thus creating a copy), we'll check that there is enough memory _then_.

That would move the "danger" of memory being yanked from underneath you. But that means that a process that calls fork() may then run into a problem some indeterministic time later.

That is kind of a killer.

  ERRORS

    The fork() function will fail if:

    [EAGAIN]
        The system lacked the necessary resources to create another process, or the system-imposed limit on the total number of processes under execution system-wide or by a single user {CHILD_MAX} would be exceeded. 

    The fork() function may fail if:

    [ENOMEM]
        Insufficient storage space is available.

The POSIX specification is clear on this: you can hard fail. Also, as an aside malloc() can also fail and the program must be able to handle that. Unfortunately for some programmers (mainly coming from GC-aware languages), they just assume that it won't fail. (Also, most rely on fork-exec but there's posix_spawn() for de novo execution).

Again, NT does indeed have a fork function, however the Windows API doesn't expose that (and this is the hack Cygwin used when forking), but it's designed to fail instead of overcommit when all memory is exhausted.

> Unfortunately for some programmers (mainly coming from GC-aware languages), they just assume that it won't fail.

In most GC languages I've used, if malloc fails for some reason (we rarely call it ourselves directly) an error happens and, unless handled, the whole app crashes. Which is fine, BTW, because I much prefer an app that crashes and dies to one that does things with the assumption something that didn't work did.

You CAN hard fail, but not overcommitting can severely limit the system in the very common case where you fork and exec. See e.g. https://lwn.net/Articles/785430/

Windows for me frequenctly needs a pagefile 8x the size of my memory in order to service the committed memory, even though it's never touched in practice.

> why Linux still needs oom-killer when Windows can run just fine without such functionality

My Windows 10 installation just BSODs when Firefox fills up the RAM if the page file is disabled.

I'd much prefer the OOM killer.

Chrome just crashes, it seems like the fault may lie with Firefox.

>My Windows 10 installation just BSODs when Firefox fills up the RAM if the page file is disabled.

I never had that happen. What actually happens is firefox just crashes, or the screen goes blank for a few seconds while the graphics driver resets.

I get all of those, too. BSOD occurs less often when the system was already pressured with several cycles of the above.

But even the graphics driver crashing is already bad enough. If there's not enough memory, offending programs should be crashing, not drivers.

(and yes, ram sticks are good and verifed with memtest86 and testmem5)

Running with the page file disabled is bad. The system runs out of address space, hence the crashes.

> if the page file is disabled

Ouch. Never run Windows without the swap. It can be as small as the system allows (AFAIR 16Mb for Win7), but it should be enabled.

While this is the obvious footgun the problem is nobody ever tests with the swap disabled.

I've seen very weird behaviour without the swap, the most funny one was when some 3D game (sadly don't remember the name) internal timer went completely nuts without the swap and game was working just fine with the swap enabled.

That's a good point, and I was actually running with 16MB page file at that time (and my reasoning for that was exactly like yours). So unfortunately it's nowhere enough.

Funny thing is I clearly remember Windows 7/8.1 showing a warning about memory running low and then killing Firefox. On Windows 10 they managed to bork that, too.

why in the blue blazes would you disable the page file?

seriously asking. if you're fiddling with that in the 21st century you are Doing It Wrong.

You're missing the main point of my comment while falling to the swap trap:

Processes may crash, but kernel shouldn't, period.

If it crashes today with no swap while running 5 programs, it'll crash tomorrow with swap enabled while running 10 programs.

a kernel panic is sometimes the right thing. "you are doing a bad thing; I quit" is a valid response in extreme circumstances. turning off the page file in Windows in 2021 is an extreme circumstance.

a requirement of the kernel is that there is virtual memory space. the page file is part of what enables that on Windows.

on Win 8.0 (I think) and before, the page file was nowhere nearly as vital, and you could turn it off in certain situations. an educated user could use their knowledge to know when it was ok to turn it off.

new versions of Windows change many things; they are not simple reskins of previous windows versions. the kernel and low level stuff changes a lot, version to version. knowledge that was valid in Windows XP is very likely no longer valid on Windows 10 or 11, and certainly is no longer valid in all situations.

keep your knowledge up to date. even things that appear to change relatively slowly, such as Windows, experience a lot of churn at lower levels. the APIs may not change, but their implementations often do.

Windows uses a dynamic page file for swapping so it will just keep increasing the page file size until it hits the limits of the OS or config settings. You can do the same thing in OSs like Linux and BSD but most distros still default to fixed swap partitions. For example macOS uses page/swap files instead of a swap partition by default. It's just a design choice. Both have advantages and disadvantages. Linux and BSD distros usually are more aggressive about kicking out applications in favor of keeping the system stable but the trade off with OSs like Windows is misbehaving applications can more easily take down the OS.

Do you have swap?

It's been a while, but I think it had 2 GB of swap on a 16 GB SSD (this was one of the Linux-friendly Chromebook models, hence the small storage). In any case, reports seem to vary considerably on how this behavior varies with swap size or presence in practice.

I think this happens when the kernel starts swapping executable images / shared libraries in and out and keeps going for a long time because according to the kernel metrics it is making great progress, even though user space is completely stalled because running code is paged out and has to be paged back in to do anything.

earlyoom kills stuff sooner instead of trying this for a long time. I’ve also heard that some people have had some success by locking base system libraries and executables in memory.

(Please let me know if this explanation is not correct or there are other solutions worth trying)

I normally run Linux without any swap partition and this still happens without swap. Realistically is this kind of oom condition (where oom killer fails to trigger) I just end up pulling the plug.

I’ve never been able to track down the exact cause.

Code can almost always be paged out, even without a swap file, because it's memory mapped from files on disk. In fact running without swap makes this behaviour worse (i.e. it happens sooner and more suddenly) because then the kernel can't prioritise keeping currently running code in memory over keeping rarely-accessed data in memory. In general you should have some swap and run with earlyoom if you want decent behaviour in a low-memory condition on linux.

Good point.

I've been using auto tab discard on Firefox. Its a small annoyance for users but if you are like me and have more than a few tabs open, this saves a lot of ram anxiety

The reason this happens is because due to memory pressure, the IO cache is emptied and thus every IO request (even if reading just one byte) is passed through to the actual storage device. This quickly fills up the device’s queue and IO performance can no longer keep up with the system’s demands even for normally-benign stuff such as writing logs.

> How about this one: the entire system becomes so unresponsive that even a VT switch takes multiple minutes to process, never mind actually logging in and running diagnostics. I don't know exactly why, but I used to get this pretty regularly on a 2 GB laptop if I did the wrong thing in Firefox.

It looks like 12309 https://web.archive.org/web/20140816132654/https://bugzilla....

There's an annoying trend in the newbie Linux world of comparing the RAM usage of different DEs and distros and using that to determine how bloated they are. It keeps annoying me because they conflate cached ram with used ram, and a desktop environment might actually use more ram but be faster in general. Pretty weird to be proud of a DE using ~500MB of ram when you have 16GB. And sometimes some memory usage is percentage based. My desktop consumes a couple GB at boot and as much in caches, but I have 64 of them.

Free memory is useless memory. You paid for it, let it work for you.

What annoys me most...when they compare it they often completely forget to remove the filecache, like the famous openbsd example. Install OpenBSD then Htop (why why why??) then be astonished that it uses 380MB (350MB of that is filecache and isn't shown as cache in Htop)

Easy fix. Just download some more RAM till you have enough.

https://downloadmoreram.com/

Unfortunately, RowHammer mitigations mean that this is not the silver bullet that it once was.

Easy fix indeed:

https://en.wikipedia.org/wiki/zram

Indeed

> wait you can actually download more ram if you set up google drive as swap space

https://twitter.com/tjhorner/status/1470788584043073542

You can always get an IBM mainframe and pay it under the capacity on demand contract.

I remember the Windows SuperFetch* memory caching service getting the same (misplaced, in many cases) complaints for a while after it was introduced on Windows Vista and 7. Unused memory is wasted memory.

* https://www.techradar.com/news/software/operating-systems/ho...

I'm regularly explain what MS SQL doesn't eats up all available memory like Cookie Monster eats cookies.

The most recent instance was just a month ago.

Though I like when I ask people to explain why their app/service should /not/ use free and abundant memory to serve their users. That short-circuits their brain.

>Though I like when I ask people to explain why their app/service should /not/ use free and abundant memory to serve their users. That short-circuits their brain.

here's a reason: because if chrome decides to eat 1GB of memory per tab, and I only have 8GB of ram, then I can't run anything else if I have more than a few tabs open. If chrome could have gotten by with 0.2GB per tab, then it using "abundant memory to serve their users" doesn't actually serve their users, it makes their experience worse. I suppose the system could work if there was some mechanism for the OS to demand that a program free up memory, but I haven't seen that in practice.

Obviously the usual context of the discussion was server workloads, often a single role servers, like a before mentioned MS SQL.

> I suppose the system could work if there was some mechanism for the OS to demand that a program free up memory, but I haven't seen that in practice.

Memory mapped files works exactly like that, IIRC.

top and free also double count memory footprint of common shared library’s across many processes that are using the same “.so” library. Use pmap to see how much unique memory each process is actually using vs. shared libraries.

Someone really needs to call this out properly for the b.s. it is. I had to dd copy very large files that caused the cache to fill up and eat ram, the whole system would come to a crawl. Copy thousands of multu gig files while using firefox, I dare the author! 22GB+ free ram and still can't handle it. It really needs a way to control n% of ram so that absolutley no vfs caching whill happen on that percentage of ram.

I wonder have anyone considered fixing that? That's an obvious usability bug of these reporting tools. They should not include disk cache in the "used" memory.

Windows does it right. On the task manager graph, disk cache is not counted as used memory. And for experts, there's a separate "memory composition" diagram, with the following parts: in use, modified, standby, and free.

>That's an obvious usability bug of these reporting tools.

Not it's a human bug (not knowing enough about your system).

>disk cache is not counted as used memory.

But it is, and people would be shocked to see that windows uses 17GB of memory after startup ;)

> Not it's a human bug (not knowing enough about your system).

I know enough about my systems, yet I think Windows is doing it right. The numbers reported by Linux, or visible on the “memory composition” diagram in Windows task manager, are borderline useless for everyone except OS kernel developers.

Despite I know these hairy implementation details about the ways OS kernel manages physical memory, I don’t really care. When I check these system tools, in 100% of cases I want answer to the question “how much ram my applications could use without swapping”.

If i go to top i want to know how much free ram i have without touching the filecache, used ram is used ram.

>Despite I know these hairy implementation details about the ways OS kernel manages physical memory, I don’t really care.

I do, and obviously everyone that implements top and htop etc.

It's neither bug nor human bug. It is a misconception of those who already familiar with windows, and most of new Linux users are come from windows.

Linux folks say that unused memory is not good, of course it is. Windows folks say that free memory is a good thing, and of course it is.

See? It actually is the same thing that being seen from different perspective.

> It actually is the same thing that being seen from different perspective

Absolutely, but these perspectives aren’t equally useful, it depends on the user and their use case.

The real low-level thing reported by Linux is a useful perspective for kernel developers, and maybe for students learning about virtual memory management in operating systems.

However, the fake data reported by Windows task manager is the useful perspective for everyone else. I think 99% of people using these tools are in this category.

The task manager in KDE (Ksysguard) does show RAM usage like Windows task manager does.

So a new user clicking System Monitor will get a RAM graph they understand.

> Linux folks say that unused memory is not good, of course it is.

30 or so years ago, while watching over a friend of mine's shoulder who was sitting at the gorgeous 3278-2 system console with the legendary beam-spring keyboard, looking into usage stats on an IBM 4381 mainframe, I noticed the CPU (I think it was only one on that machine, that occupied most of a floor) was pegged at 100% usage. When I asked if that was bad, he simply responded:

"Of course it's 100% used! We paid for it, we'd better use it!"

Later I understood that he'd probably be more concerned if the IO processors started to get saturated, but, by then, I was no longer using mainframes.

I love single purpose websites like this this. Nice work whoever made this.

Indeed. It follows the philosophies behind best/better/etc. motherfucking websites, very lightweight, and works without javascript.

I've hit heavy swapping cases a few times on my home linux box and was dissapointed in oom killer. I wanted it to kill something so I could get my system back, but it didn't seem to trigger. Instead the system became progressively more unresponsive until basically all UI updates stopped. I tried switching to a console prompt but even that was like molasses. Does it only kick in when the swap is so full? If so then perhaps it's a bit late as things so down so dramatically as swap nears being full, it may not be able to become full as the system grinds to a halt. I'd be interested to hear what experiences others have had with oom-killer

>Disk caching makes the system much faster and more responsive! There are no downsides, except for confusing newbies. It does not take memory away from applications in any way, ever!

>You can't disable disk caching.

I guess I'm lucky I've never been confused by this. All the RAM monitors I've been using (KSysGuard and conky) must be reporting RAM usage accurately.

Yes, Ksysguard does hide cache usage.

Reply on news.ycombinator.com