Hacker News Reader

> My experience with it is that once a disk-backed machine (as opposed to SSD) has started swapping, it's essentially unusable until you manually force all anonymous pages to be paged in by turning off swap ("sudo swapoff -a" on Linux) or reboot.

That depends. If your workload exceeds the amount of available memory, you will start "thrashing" the disk and that can make a system un-responsive.

If you happen to launch a large application, or start working with a big file, unused pages will be evicted to disk to make room and, after some slowdown, the system should become perfectly usable again. YMMV

On OSX, I don't know a way, but I can't recall the last time I had to reboot due to RAM/swap issues, even when I was developing apps on a 4GB Macbook Air. I guess memory compression, which is enabled by default, helps here. Most OSX systems have very fast SSDs as well.

On my 2015 MBP with 8GB & SSD, I am often stuck for 10-15 minutes unable to do anything while thrashing. And I am someone who has Activity Monitor handy. I do not have this on my much older and weaker Ubuntu X220s doing the same type of development. Not sure why that is.

If it's 3rd party SSD, have you enabled TRIM? I had to do that for my old Mac Mini, made big difference. (2015 MBP of course has factory-installed SSD, but maybe this helps someone else.)

http://osxdaily.com/2015/10/29/use-trimforce-trim-ssd-mac-os...

It's all original Apple hardware.

It can be Spotlight reindexing, or Time Machine creating a backup into /Volumes/MobileBackups (done once per hour). Especially taxing if you have lots of files on your disk.

It's kind of annoying that these processes aren't niced and ioniced (or the macos equivalent). Especially when typing a password.

Look for hung mds (spotlight indexer) threads. They can get hung up and fail Un-gracefully with some files.

I had a client whose app generate PDFs that would cause this to happen.

Memory (8GB) is just suddenly jumping to red, usually by some Safari tab (gmail etc) which suddenly jumps to 1GB. And that's it. No indexing or mds processes; just browser tabs which suddenly jump over a point and lock up everything because of memory use. It's like i'm back in the early '90s when I first touched a non Amiga and non SunOS system. And asked how people can work with that 'Windows/DOS stuff'. I am not sure why it misbehaves so much though...

Usually when people say 'swapping' they mean page faulting. It's nothing more than a slight annoyance on a single-user machine if you swap for 10 seconds, but on a busy server you are dead in the water.

Take this with a grain of salt since this anectode is a few years old, before I upgraded to a SSD to host my swap, and this is on Windows.

I'll always remember when I used to load a large piano instrument in a VST DAW on Windows 7, taking about 3-4GB out of 12GB of RAM, it played perfectly fine but if I left the application open, invariably on the next day I'd get a barrage of audio dropouts when pressing any new piano key. One trick was to put my arms on the entire keyboard a few times to wake the swap back up to memory. Another trick, which I ended up relying upon despite occasional low memory warnings, was to disable the pagefile entirely - that sure fixed the problem.

I'm not sure how/if things have improved since with Windows 10 and SSDs, but I always felt there was something wrong with the algorithms, since even with GBs of memory free at all times, old memory content would tend to end up on disk, without any good reason I could see.

I assume the OS used time to prioritize various caching/pre-fetching techniques over actual application data, and/or once paged, never preemptively loaded data back to RAM even if plenty of memory was available.

> If you happen to launch a large application, or start working with a big file, unused pages will be evicted to disk to make room and, after some slowdown, the system should become perfectly usable again. YMMV

What is an unused page? One that the foreground, memory-hungry application doesn't need? Okay, fine, but what happens when you switch back to some other application? My experience is that it needs the RAM that was paged out, and it doesn't get paged back in all at once. Every time you hit some 4 KiB of memory that happens to be paged out, you wait another 10 ms. I don't know how much beyond the 4 KiB gets paged in at the same time. Worst-case, there's no read-ahead at all. Let's say the application is using 1 GiB of RAM. Then this can happen 262,144 times, which means 44 minutes of waiting in small bursts as you're trying to use it, rather than the 10 seconds (at 100 MB/s) it'd take to read it all in one go. That's what I mean when I say the machine is unusable.

This is my experience too, and this thread has motivated me to "swapoff" all my desktop systems today I think. There's no situation in which swap usage for me has ever not lead to a reboot due to the system going unresponsive the moment it starts swapping.

I will posit that your swap is too big then; I have 16GB of ram and run with 1-2GB of swap. A runaway process hits the OOM in about 30s.

To be fair, this is probably ZFS-on-Linux having a bad swap interaction. But I have found that once I get to 95% RAM use, I start hitting the swap fairly frequently and as a side-effect my desktop starts stuttering (usually happens if I kick off an 8-core code compile).

The other culprit I suspect is too many potentially blocking processes Cinnamon's main thread, but I've never figured out a decent way to go after the problem.

oh ZoL is terrible with swap; I get OOM all the time when my ARC is still like 4GB.

"My experience with it is that once a disk-backed machine (as opposed to SSD) has started swapping, it's essentially unusable"

The OS should start swapping very early to avoid bursts of disk I/O and rendering the system unusable. On linux this is somewhat configurable, even if not user friendly, but a combination of swappiness and vfs_cache_pressure could turn it into a usable machine, taking care of inefficient memory usage, memory leaks, unnecessary vfs cache, etc.

> The OS should start swapping very early to avoid bursts of disk I/O and rendering the system unusable.

I think you're talking about the I/O of paging dirty things out, but I'm talking about the fact that some memory location is no longer present in RAM, so accessing it will take 10 ms or more to page in.

The system is not only useless while actively swapping. It's useless after it has ever swapped, and you can only recover by disabling swap ("sudo swapoff -a") or rebooting.

Contrarian anecdote: I've recovered from swapping a few times on a desktop machine with an HDD (typical scenario: an ON clause was omitted from a join and postgres is doing a cartesian join between two tables). I didn't find things unable instantly, I was able to recover by SIGTERMing the relevant process and then running `swapoff --all` while maybe going for a tea break. YMMV.

On OSX it used to be that you had to disable the pager daemon, then remove the swap file. Probably need to disengage system protection for it to work though.

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

Then:

rm /var/vm/swapfile*

Disclaimer: haven't tried doing it this way on Sierra.