Comment by scottlamb | Hacker News Reader

Comment by scottlamb | Hacker News Reader

This is bad advice when the swap is backed by spinning disk rather than SSD, in large part because paging in once the problem is solved is not aggressive enough.

Let me make it concrete: say 1 GiB of your application's RAM gets swapped out then is later paged in only as needed (4 KiB pages) with no readahead. Now the application's VM space is a minefield: there are up to 262,144 times it can stall for a ~10 ms disk seek (for a total of ~40 minutes). Sequentially reading in 1 GiB of RAM from disk would take only ~5 seconds.

Hopefully OSs use some readahead, but I'm sure Linux and macOS don't use enough. I find a system that has ever swapped to be totally unusable until I do "sudo swapoff -a" (on Linux) to force everything to be paged in, or just reboot (on macOS, I haven't found any other way).

Some of this can happen even without swap: the OS will still drop clean file-backed pages, so unless you mlock() your executables after startup (my production binary at work does this), you can still have major page fault-induced latency spikes.

Swap backed by SSD (or compressed RAM) should be more reasonable, but I've had enough bad experiences with swap on spinning disk that authoritative-sounding articles that encourage swap without mentioning this problem piss me off.

In contrast, if you don't have swap and run out of RAM, something will die and get restarted. In many cases this is a much better failure mode than continuing to run slowly.

Also bad when it's backed by an attached drive from the cloud provider, like an AWS EBS volume -- it's particularly nefarious if you swap to the root drive, swapping can exhaust the EBS burst balance, and instead of 3000 IOPS to the root drive, after swapping sets in you're down to 300 IOPS (for a 100GB drive), and performance really slows to a crawl.

I know I'm giving up some RAM that could be swapped out without impact but it's worth the stability of never swapping to an attached drive.

I've had the opposite experience on Windows. I always put swap on HDD, because I've been bitten several times by memory leaking applications rendering the entire OS unusable until the SSD with the swap file ran out of space.

Windows would happily grow the pagefile beyond the limits I had set, and if the allocation loop was tight enough the OS chocked on handling all page faults. As in using literally minutes to respond to keystrokes.

By putting swap on a slow HDD, Windows has plenty of time to service mouse and keyboard input while waiting for the disk to respond, allowing me to kill the offending application before saturating my disk.

I assume this behavior was OK back when computers had very limited memory, and disks were slow anyway. But these days, if a single application wants more than the available memory (physical + pagefile), I'd rather the OS just said no to the application.

I don't use Windows, but it sounds like you can't you set limits on how large the swapfile can get? That's crazy!

> But these days, if a single application wants more than the available memory (physical + pagefile), I'd rather the OS just said no to the application.

I agree.

Oh you can set limits. And then Windows will promptly ignore them, at least under certain circumstances. Why I have no idea.

Swap sucks, but OOM killer sucks more. It certainly isn't a great solution, but it's good for temporary spikes.

You can adjust "swappiness" if it's moving stuff to disk too aggressively.

> Swap sucks, but OOM killer sucks more. It certainly isn't a great solution, but it's good for temporary spikes.

I disagree. The OOM killer usually kills the right thing, and if it doesn't a quick restart is usually better than continuing to run slowly.

You can also just put things in containers and set per-container limits.

> You can adjust "swappiness" if it's moving stuff to disk too aggressively.

What I want is to adjust un-swappiness: aggressiveness of paging stuff back in. Once the memory spike is over, it should page everything back in automatically.

The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen.

Containers don't really solve your problem, it just adds a layer of indirection between the processes hogging RAM and the OOM killer, which in my experience adds even more uncertainty into what the OOM killer is going to do.

Tuning oomkiller's hitlist makes it much more managable and the new BPF tools to track it make it a ton nicer than a system grinding into swap.

> The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen.

I won't say there's no circumstance in which the OOM killer is worse, but in general I disagree with you.

A stateful server application: maybe a standard ACID DBMS like PostgreSQL. I'd much rather it get killed (failing whatever transactions were in progress) and restart in a healthy state than continue to run slowly. Here I'm relying on a correct implementation of Durability, but if you don't have that, you have other problems.

Even more so for a quorum-based database server in which the leader dying (and another taking over) is much better than continuing to run slowly.

And yes, even more so in stateless servers, particularly when there are several of them. Going down and restarting quickly (a brief loss of capacity, a brief spike of errors if the client doesn't retry) is a lot better than continuing to be slow (a sustained loss of capacity, possibly leading to cascading overload; and user-visible latency problems if there's no hedging).

For something like a desktop application, it's debatable, but personally I'd still rather endure the up-front nuisance of having to restart it than have the OS try to paper over it until I finally realize why it's so painful and do something about it. (Insert anecdote about frog boiling here; I hear frogs actually do jump out before they boil, but yet somehow the story still rings true, if you know what I mean.)

Databases are just one kind of stateful server application though, and yes I would agree with you for that usecase relying on it's underlying durability is a good thing, because it's built with that in mind.

Instead I would like to point out the ubiquity of things like nodejs, which make it all to easy for under-experienced developers to shoot themselves in the foot w/ a chaingun with respect to data design and management. It's not really the internal state of the nodejs process itself, it's the external state that it is inevitably being manipulated in the node process.

I'd rather go full chaos monkey and shake out such bugs (/design flaws) than try to avoid them.

Completely agree there.

Some times that luxury doesn't exist though and you just have to get shit done and hope playing fast and loose doesn't come back to bite you.

Like all things in life, it depends. If you have a truly massive server, then swap becomes infeasible, and the OOM killer is an absolute lifesaver. In fact, so much so that you really want the OOM killer to act earlier, to prevent the system getting too low on free RAM/cache. There's a program called earlyoom that I installed on our server with 384GB of RAM - it has been useful multiple times, because it prevents the system entering the thrashing state that would make us basically email IT support and get them to press the big red button on the front.

Secondly, the thing with swappiness is that it alters the balance between stuff that is cache and stuff that is live data. But "stuff that is cache" includes your running software text, libraries, essential operating system bits, etc. If you set swappiness too low, then in a low memory situation the OS will throw away all the code that it is trying to run instead of swapping out some of the live data. That is what many people don't realise.

Author said almost always. Nothing is absolute in cases like this. I agree with the article. We enable swap on our production servers precisely because the engineers don't rectify their memory leaks or whatever problems are internal to their software. Some of those engineers are our own and we can't convince them to fix their software. Others are for things we have no control over, such as Kafka, for example. When Kafka starts going out of control and you've got no swap, oom killer will kill it and you'll end up with corrupted index files, which is a bigger problem than the performance problem you'll see if it has to swap for a while until it gets itself sorted out.

Reply on news.ycombinator.com