Tired of explaining Linux's seemingly obtuse memory usage to confused newbies? | Hacker News Reader

Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

linuxatemyram.com

155 pointsyankcrime70 comments

This is misleading, as the kernel will often swap out processes rather than shrink the disk cache. Its tendency to do this is controlled by the vm.swappiness parameter, which by default heavily favors swapping: http://kerneltrap.org/node/3000

Decreasing this value will favor reducing the cache size and will tend to increase responsiveness. You can set it in the /etc/sysctl.conf file. Low values (≤ 5) are especially good for SSDs where you want to minimize swap usage.

I often just turn swap off entirely. A normal workload on my 4GB linux machine will frequently fail to fill even 2GB of buffers. At the moment my machine has still managed to fail to fill 800MB of my RAM with anything and I'm running two virtual machines and doing my usual work load.

I do have a swap partition but only so I can hibernate.

> I often just turn swap off entirely.

And you're smart to do so. Swap is useless for 99% of end-user systems. ChromeOS doesn't use any swap partition, for instance.

As for hibernation, it's not available if you use a crypted swap, which is wise to use (at least on a laptop).

As a result, I, too, disabled swap completely.

"And you're smart to do so."

Finally, that's the first comment I get, instead of someone extolling the virtues of swap and babbling warmed-over 1990s rules of thumb about "twice the RAM is the recommended size of your swap file" as if I'm going to wait for even 200MB of swap to fill up before flipping out and killing the offending process, let alone 8GB.

>Finally, that's the first comment I get, instead of someone extolling the virtues of swap and babbling warmed-over 1990s rules of thumb about "twice the RAM is the recommended size of your swap file" as if I'm going to wait for even 200MB of swap to fill up before flipping out and killing the offending process, let alone 8GB.

Those people have doubtless misunderstood the point of swap. You should have a swapfile/partition because it allows allocated but currently unused memory (from an application which keeps data hanging around which is not needed for most of it's working life, or an application which simply leaks) to be dumped to long term storage, thus freeing memory for its real use: page cache. Sweet, sweet page cache.

I'm always happy to see a few tens, even a couple of hundred MB of swap in use, because it means that some application had some unused data hanging around for so long that to leave it there would mean my machine having to read from disk more frequently, which would be Bad.

> "twice the RAM is the recommended size of your swap file"

Don't listen the parrots that repeat something that ceases to be true at least 10 years ago. Swap used to be a useful hack, it is not anymore.

> as if I'm going to wait for even 200MB of swap to fill up before flipping out and killing the offending process

Well said ;-)

Hibernation requires a large swap space, doesn't it? Last I knew it did, perhaps you can compress it now.

Yes, and it may seem a merely semantic difference, but there's still a difference between "swap space" and "hibernation backing". One I'm willing to wait for while it fills, the other, I am not. The kernel may not distinguish, but I do. When I don't care about hibernation I just remove it.

Right, swapoff is totally acceptable imo if you're having trouble with swap. I was commenting on how "double the size of RAM" is a bad or outdated guideline. There's been a few times where I've regretted not making my swap space big enough, sometimes when getting a RAM upgrade and wanting to do hibernation, etc. My disk isn't super pressed for space so in my mind there's no real reason to be stingy, and then you don't have to grow/shrink partitions if an upgrade occurs after the initial disk setup.

Swap still provides one important function: it allows large, inactive, long-running processes to be moved from RAM to make room for more caches and buffers.

That said, if you never fill your RAM with buffers/cache, then of course swap makes no sense.

On the other hand, one could say that moving a large process from RAM to swap certainly makes it nearly inactive and long-running...

Memory is cheap. I'd rather pay a little more and have the long-running processes stay in memory than worry about my more active processes ending up in swap accidentally. If I wish to reserve 1GB for these long-running processes in RAM instead of swap, I can still win by buying 2GB more RAM.

Besides there's always some disk-backing you can't generally avoid: pages containing read-only executables can be purged from memory when unused and re-read from the original .so or binary when needed. This is something that would never go to swap anyway.

Care to explain why this is so?

Stop measuring your swap space in terms of disk space, start imagining it in terms of "amount of time it takes to fill up". At a nice round 20MB/sec write, it would take a solid 400 seconds to use that much swap at full speed. In reality, you can't fill it that fast either, because it's seeking, also yanking other stuff out to run your other processes, and then sticking the stuff back in to run those other processes, so it's really a mess; the real amount of time it could take just to make initial real use of 8GB of swap could be days, no joke. (In other words, I don't care if you have a 80MB/s drive, it really doesn't matter much, plus swap usage tends to be full of seeks anyhow so it's not like you're going to get 20MB/s either.) Given that a modern system is, relatively speaking, brought to a near-complete halt by being in swap, what possible process are you going to run where you are willing to put up with your system being brought to a halt for even tens of minutes at a time, let alone the hours or days it'll take to fully utilize 8GB?

Clearly, you don't need and basically can't use an 8GB swap partition. So, how long are you willing to put up with? That will vary, but let's say 10 seconds before you're "flipping out and killing processes". That's a 200MB swap partition. But... that's only 5% of the size of your RAM! If that's the difference between a process completing or not, you've probably already lost. Or you should just kill Firefox.

On a 4GB system, the most likely reason a process is pushing you into swap is that it is in an infinite allocation loop, and all having 4GB of swap does is make your system crawl that much more before the process dies.

This is a result of RAM sizes increasing far faster than hard drive write speed has. When I had 32MB of RAM, it made sense to have some swap. I could swap out, say, 16MB of unused executable pages (bits of windows, bits of drivers I'm not using, bits of the massive Office suite I'm not using, etc) and get that much more working set, and this could happen in a reasonable time; the system choked for a couple of seconds but recovered in a stable manner. Now swapping out 16MB of executable is a joke. SSDs may change the balance, but these balances have been out of whack for a long time, I rather suspect that even with an SSD it won't be worth swap. Especially since by the time SSDs are truly common 8GBs of RAM may be entry-level because, well, why not? (Poking at Best Buy really quickly, at the $500 line you get 4GB for laptops, a little over $300 for desktops, coming down fast. I'm not sure they have anything less than 2GB now and even that is really into "don't use swap" for the average user.)

Please note that this doesn't apply all the same to servers. I have servers running with 16 GB of RAM and 16 GB of swap (because I simply can't even stop them to add RAM), and even then about twice a year they run out of memory and the OOM killer does its dirty job. However without swap they used to simply crash and burn, so swap is fine in this case.

The only thing I can possibly imagine that a server can do with 16GB of swap is leaking garbage like a sieve and then getting the garbage swapped out. On an incredibly local basis, yes, this might be a better idea than not having swap, but as a generalized reason for servers to have swap it's terrible. If that's not what's happening I am all ears as to what the situation really is, but I'm sure it'll be something very unusual; in general if your server has even the slightest need for performance it can't use swap.

Not a memory expert myself, but a friend of mine dismisses overly large swap as consuming more RAM to handle it's addressing - that is, the more swap you have, the less of your RAM you can access.

I'm not sure what the consumption rate is, but he won't create general servers with more than 2GB swap (if that), no matter how much RAM the system has.

You're probably never going to hit the swap for a good reason. You might hit it if you do a mistake (resizing an image by 5000% and not 500%), in which case you will suffer if you let your gigabytes of swap get filled), or if you have specific needs (video editing, and still...).

Keeping a swap partition in a 2011 computer is 1. a waste of disk space, 2. an unecessary matter of worries (2-1. may leak some infos, even if your other partitions are encrypted, if your computer is stolen, 2-2. a source of potential bugs).

Swap is mainly an emergency backing, i have nagios message me whenever swap hits x amount depending on the server, it's better than oom killer picking off processes.

It might be better to use proper rlimits, and not let a mad process gone wild triggers the OOM. Swap just delays the problem.

The problem with set number limits is that changes in software and hardware require changes in limits - it's easier to just watch swap and get notified.

Is there a way to decrease the swappiness of OS X?

The article mentions this on the second page at the bottom: http://www.linuxatemyram.com/play.html

"But look you found the notice didn't you?"

"Yes," said Arthur, "yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of The Leopard'."

Low swappiness is also useful in tightly controlled VPSes. Swap is absolute murder on a VPS platform. I leave it on as emergency backstop, but otherwise I have turned down to 0. I simply do not want processes swapped out, ever.

You can't disable disk caching. The only reason anyone ever wants to disable disk caching is because they think it takes memory away from their applications, which it doesn't! Disk cache makes applications load faster and run smoother, but it NEVER EVER takes memory away from them! Therefore, there's absolutely no reason to disable it!

Never say never. You can and should disable/purge linux buffer caching in certain situations. See O_DIRECT and/or POSIX_FADV_DONTNEED. If I'm writing a database, I take care of my own caching, I do not need the OS doing it as well. Also, if I'm doing a huge rsync of a directory tree, just because I just read every byte of every file in that tree, I most certainly do not want to load all of that crap into the linux buffer cache. It will evict my already warm set, as well as potentially cause a ton of swapping.

Funny, I just noticed this in the Mac OS X rsync man page for the very first time:

       --cache
              Apple specific option to enable filesystem caching of rsync file
              i/o Otherwise fcntl(F_NOCACHE) is used to limit memory growth.

There was a patch for all architectures, but it got dropped after 3.07 (current is 3.09). Andrew wrote "I don't feel this patch is general enough nor straight-forward enough to make it into rsync."

See these for more: http://insights.oetiker.ch/linux/fadvise/ https://bugzilla.samba.org/show_bug.cgi?id=7004

A good way I read somewhere was "Unused RAM is wasted RAM. Linux uses it to speed up your disk and gives it back to programs when they need it".

The title of the link (unlike the linked page) suggests that Linux is somehow to blame here... Why? Using RAM for disk caching is a good thing.

Besides, this technique is employed by Windows as well. It's just that the average Windows user probably doesn't know how much free RAM his PC has (well, the really average user doesn't know how much total RAM it has). My Windows 7 box takes at least 2GB of memory to itself for disk caching, AFAIU.

Windows started doing disk caching with Vista and users certainly noticed and complained in much the same way.

Windows didn't start doing disk caching with Vista. Perhaps it used memory more aggressively, or changed the way it did the accounting, but it certainly didn't start doing disk caching just with Vista, as anyone who remembers the difference in performance of DOS before and after running SMARTDRV will attest.

2000/XP was always extremely aggressive about allocating memory for cache. It would very often swap stuff out to increase available cache, which made the apps you did not use for just a few seconds completely unresponsive.

Specifically, minimized applications got their working set trimmed. You could see this in Task Manager: it's Memory Usage column was actually Working Set (I think they're more explicitly labelled these days) and it frequently went from many MB down to a few KB once you minimized the app. Working set is the set of pages, currently in memory, that the app has used "recently", and are thus less likely to be paged to disk. Shrinking the WS makes pages more likely to be paged out.

Windows started doing pre-emptive caching with Vista, based on the applications you tend to run.

I think the average user doesn't even know what RAM is. My neighbour, for example, doesn't know the distinction between the space available in RAM and the capacity of his harddisk. Which led to a strange misunderstanding when he told me his PC had '250 giga of internal memory'. I tried to explain than with 'internal memory' people usually mean something else, but I didn't think he understood. Never mind, he doesn't grasp hierarchical filesystems either, but all is fine as long as he can edit his photos (which in his perception are kept within Elements) and 'surf the internet'.

My apologies, that's certainly not what I meant to imply (and hence the use of the word seemingly). The point I was coming from was with regards to the number of times I've had to explain these concepts to an exasperated friend or cow-orker, when they've exclaimed something along the lines of "I've just looked at `top` and there's no free memory!".

This paragraph: 'But what do you call memory that is both used for something and available for applications? You would call that "free", but Linux calls it "used".' is IMO the best way I've ever seen of explaining the issue. It also reveals the solution: call it free! It is free, after all. Nobody cares about how much free ram is cache unless they are doing something really specific, in which case they should be passing specific flags, or using a specific tool, to report it.

Unfortunately there is no chance Linux will do this, as the relevant reporting tools are too entrenched (and with good reason).

Or just rename "free" to "wasted". I usually tell people the free number indicates how much RAM they paid for but aren't using.

It is admittedly difficult to re-learn what you know about computers, and what Windows has taught people (including myself) is that RAM usage must be below ~75% for optimal performance. It has been my experience that paging becomes much more blocking above that.

Maybe add a new (end user-friendly, preferably graphical) tool to report memory usage the user friendly way? Then vmstat can be left as is, and end users can be directed to use the new tool.

By the way, this behavior is not Linux-specific. AIX and other Unices have been doing this forever. And end users (who should have known better, BTW) would complain too. The only machine that I remember as showing a sizable percentage of the memory as "free" was a just-imaged top of the line server. Half a gig or RAM, and almost all empty! -- this was the early '90s.

Edit: Just noticed that graphical tools that report memory usage in user-friendly terms already exist (gnome system monitor). My bad, I always use vmstat and sar myself, and only recently found out about /proc/NNNNN/status

Hmm, this situation is like owning an electric vehicle. The manufacturers are passionate about technology, and they invented a flywheel that can store massive amounts of energy, giving your vehicle incredible transient acceleration response. They ship many vehicles with this new feature, but unfortunately, they forgot to account for what this would do to the battery charge display after starting. As a result, their reps keep on having to explain that the vehicles are not rapidly leaking charge shortly after starting -- instead that energy is being placed in the flywheel where it can greatly increase the car's responsiveness.

Whoops!

The difference between the scenario above and Linux's disk cache? Techies blame the former on the car company but the latter on the end-users.

If only electric cars were free like Linux.

My analogy has more to do with approaches to UX than it does with economic model.

Wouldn't it make more sense to just include the RAM used for disk-caching when calculating free memory? After all, the whole reason it's being used for disk-caching in the first place is that it's free! The Linuxoids that know what this even means can always explicitly look for the line that says how much memory is being used for caching...

It costs time to zero pages; disk cache pages are not zeroed. Not having a source of zero pages will add latency to large memory allocation requests.

(The pages must be zeroed to avoid the security problem of snooping on data you might not have access to.)

Exactly the kind of thing users not familiar with the way Linux handles memory DON'T consider when they check how much free memory they have. If you're the sort of person concerned with the overhead of zeroing pages then you can look at more detailed memory-usage info to find the details you're looking for.

The Windows task manager has a separate field called "Available". Right now I have 8190 MB total, 3627 cached, 4151 Available, and 537 free (not sure if that adds up perfectly, it was changing while I typed it).

OSX's activity monitor does something similar, it has 4 statuses: Wired, Active, Inactive and Free.

Wired and Active are basically the RAM being used (Wired is a special activity status for stuff which can not be moved to disk), Inactive is the cache (generally programs which were used recently) and Free is… well, free.

When checking used/free, Inactive and Free count towards free, Wired and Active towards used (though that's not always consistent).

Also, I don't think you have 8TB RAM.

If you really want the details on OS X, see http://developer.apple.com/mac/library/documentation/Perform... and http://developer.apple.com/library/mac/#documentation/Darwin...

See http://www.usenix.org/event/usenix2000/freenix/full_papers/s... for information about the BSD unified buffer cache.

And see http://www.westnet.com/~gsmith/content/linux-pdflush.htm for information about the Linux page cache.

He probably has 8GB of RAM (8192MB); some may be lost to address space holes. I have 12GB, if you're finding that implausible.

> He probably has 8GB of RAM (8192MB)

Yes, I'd expect that, but at the time of my comment (before his stealth edit) all his units read "GB" not "MB". I wrote "8 TB" for a reason.

> I have 12GB, if you're finding that implausible.

I have 8GB RAM in my laptop, why would I find 12GB RAM implausible?

It looks like gnome-system-monitor reports the "correct" amount of free RAM. I'd imagine most newbies would use that rather than popping open a terminal and typing "free," right?

You'd be surprised. Some users who use Linux consider themselves to be pretty adept, and I can easily see this problem being an issue for people who perceive themselves to be clever.

Heh, nice site.

Note I find this useful for showing how much RAM apps really are using: http://www.pixelbeat.org/scripts/ps_mem.py

    free -m | awk 'NR==3 {print $4}'

I have had the same problem, plus some, when trying to guesstimate memory usage in vps servers. The caching is transparent, so free always reports zero swap. ps gives sensible numbers, but the total used memory is neither RSS nor VSZ summed. I've heard about library memory being double-counted, but I've found no way to accurately get the numbers. In the end I pretty much gave up and settled for the settings that worked.

tl;dr: on vps things can be even uglier.

All this does is make me regret paying more for extra memory... With a dozen programs open, I am sitting at 2.3GB free, according to this. I guess it was wrong to splurge for that extra memory.

So basically the same thing Vista was doing (prefetch, and admittedly it did a shitty job with >1GB ram), except since its Linux, Oh its all fine guys, we won't destroy the OS' reputation.

The page is about explaining a correct behavior, not justifying some bogus memory usage. There's no reputation at stake. All modern operating systems are expected to behave that way, free RAM is wasted RAM.

Right, the only thing wrong with Windows Vista was its RAM caching strategy.

Maybe not, but there were a lot of people needlessly going bonkers over the disk light blinking (SuperFetch) and free memory numbers in the task manager(exactly the same problem that the referenced site is trying to explain).

See http://www.codinghorror.com/blog/2006/09/why-does-vista-use-...

I dare say Vista did a plenty good job of destroying its reputation without anyone batting an eyelash at this.

Linux has been doing this for a lot longer than Windows has. (Which I only actually know because there was so much FUD about Vista's pre-fetching when it came out.)

edit: Sorry, getting prefetching and SuperFetch confused.

MS shipped SMARTDRV with DOS 4.01, which shipped in 1989. By my reckoning, Windows (running on DOS) has had disk caching since before Linux existed.

Unix has been doing it... You talk as if Linus personally invented the idea of caching.

Doing what? Prefetching? It has been there since XP.It is not just simple caching. Windows probably was doing caching even before Linux existed.

What Vista does is SuperFetch and I don't think it's there in Linux even now, except that someone wrote a program that would do a similar thing but it's not really used that much.

Prefetching is not the same as caching. Vista's SuperFetch is even more different.

How does this compare to darwin's memory management?

It's merely a matter of definition - whether to include the disk cache in the "used" memory or not.

I love explaining it! The response is priceless, "Wow, that's pretty cool!" "No, that's Linux."

Reply on news.ycombinator.com