In defence of swap: common misconceptions (2018)
chrisdown.name
chrisdown.name
A few days ago, I got a complaint about slow ops in a Ceph cluster, and one of the Ceph OSDs spent quite a visible amount of time in response to a simple "ceph tell osd.13 ops" query that is a natural first step when debugging slow ops. Upon investigation, it was found that one of the nodes had a significantly higher load average (50) vs others (14-18). In "top", out of 256 GB of RAM that the node had, approximately 100 GB were cached and thus formally available, yet, out of 8 GB of swap, 6 GB were used, and kswapd consumed a non-negligible percentage of the CPU time.
The kernel is some 5.15.x Ubuntu kernel; the root disk (which also holds the swap file) is a cheap SATADOM, and SATADOMs, in general (yes I know there are exceptions), have miserable performance and low write endurance; I forgot to check SMART statistics but cannot exclude that the drive might have started failing. I did check the dmesg, though, and there were no I/O errors.
The problem has been resolved by disabling the swap file. I should probably have set up swap on ZRAM instead, but a configuration without swap also works.
Reading that, I just realized it could be done over the network - that would be impressively glacial. Should be a good laugh.
Heck, why not over a db9 null modem slip connection? Start a compile and send it off to the Long Now Foundation - set it up next to John Cage's ASLSP organ.
Also I don't have fancy money. I was imagining the $20 gear on Amazon/eBay. Maybe that's my problem...
Yes, those have always been impressive and usually cost as much as a luxury sports car.
Hopefully the increased demand from the AI world will help bring those prices down.
-- Configuring the operating system with swap to provide additional virtual memory for daemons is not advised for modern systems. Doing may result in lower performance, and your Ceph cluster may well be happier with a daemon that crashes vs one that slows to a crawl. --
If a single ceph node crashes and restarts, I assume that the cluster will pick up the slack and heal/deal?
the OSD's main job is file serving, so you need to use as much ram as possible to act as virtual tiered storage (caveats apply) so having a large swap to hold useful but not often used bits of ram is really useful for performance.
Our file servers used to have ram+~30% as swap.
It sometimes helps with defragmenting memory, memory pressure in general, and depending on your workload and other things running on the box, you could have better performance by being able to have more things in page cache.
But I think this ignores the likelihood that I want this particular memory reclaimed. It is not equal for files and anon memory. At least on my systems, if anon memory is dirty, it's because I did something with it, and if I haven't freed it, that means I'm still working with it. But there's all sorts of stuff that just kind of ends up in the disk cache, which I will never touch again. It is irrelevant to me that refaulting a swapped anon page is no slower than rereading a file, because the probability that the first will happen is so much greater than the second.
For all that systems try to guess about which pages are in use, or will be used again soon, I find they pretty much all get it wrong and as a result the system gets janky, and it's best to simply weight keeping anon memory in core much higher. I expect disk access to be slow when I read a file. I do not expect ctrl-tab to be slow, ever.
I think ctrl-tab might cause a disk read even without swap, because some/all of the binary of the program you're running might have been evicted from memory (backed by the program file, not swap) and need to get faulted back in when you hit ctrl-tab. The disk read might occur even without eviction, if that part of the program had never been read into memory yet.
Please realise that things that can be evicted from RAM include the code and static data for the programs you are running. If you switch off swap, then you are disqualifying stuff like data pages that a program will set up on process start and then never touch again, which is sometimes a distressingly large amount. That then uses up RAM that could be used to provide a more responsive system.
The ideal contents of swap are those data pages that are never going to be touched again. On a normally running system that doesn't have massive memory pressure, that is basically all that will make it into swap, and that's fine because those genuinely are pages that you won't need, and they aren't causing the problems with alt-tab.
What probably is causing the problems with alt-tab are one of two possibilities. Either the system is over-burdened and it is having to swap out large amounts of working set (to which the solution is to install more RAM or reduce the load, not switch off swap), or you have been transferring large amounts of data through RAM, causing the disc cache system to evict parts of a program you haven't used for a long time in favour of these files that another program keeps loading in (and switching off swap won't help much with that either, because the system is being quite happy to evict parts of that program that aren't anonymous pages - a solution is to get the program that is handling lots of data to ask the OS kindly not to cache certain things, if it for instance knows that it will only load them once).
I do think that Linux's behaviour has improved massively over the last 10 years or so. A lot of these myths about swap were created before the improvements, and we should stop believing them.
My conclusion is: In production, in a datacenter, when code is stable and compute-per-dollar efficiency matters? Yeah, sure, I can believe that swap makes sense.
On a dev machine? Not on mine for sure. If you think swap is a net positive on a dev machine, then try pasting the following (wrong) code in a bash prompt on Linux:
echo '
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
int main()
{
for (int i = 0; ; i++) {
char *mem = malloc(1 << 30);
if (mem == NULL)
return 0;
memset(mem, 42, 1 << 30);
printf("%5d GiB\n", i);
}
}
' | cc -O3 -o crash_my_laptop -x c -
./crash_my_laptop
[1]
https://news.ycombinator.com/item?id=40582029
https://news.ycombinator.com/item?id=39650114
https://news.ycombinator.com/item?id=38263901
https://news.ycombinator.com/item?id=31104126
https://news.ycombinator.com/item?id=29159755
https://news.ycombinator.com/item?id=23455051
> We can discuss the results in a couple of hours, when you recover control of your machine. (Note: with no swap and no z-ram, it crashes after 10 seconds; no side effects.)
That might be true for this contrived example. But my real-world experience is exactly the opposite.
In a case of memory over-usage (in my case because of working in a huge bazel project and several browsers open + some electron apps):
- without swap, the system at one point just got immediately so incredibly slow, that the only thing I could reasonably still do was to restart my machine, while
- with swap, the system (at higher memory usage) just got noticably slower and I could close some app / the browser to get into the normal usable regime again.
In case of memory over-usage:
With swap: system becomes unresponsive
Without swap: OOM kills offending process
This has been my real world experience 100% of the time on Linux.
And I have time to think about better problems than "You can achieve better swap behaviour under memory pressure and prevent thrashing by utilising memory.low and friends in cgroup v2."
The core problem with swap is that nowadays you need gigabytes of swap to actually make a difference (when you have 32GB of RAM, 1GB of swap is 1/32 of a difference). And you certainly don't want to system to swap in and out gigabytes of memory.
Why though?
Swapping 32GB of memory in and out of an NVMe SSD is much faster than swapping 1GB out of a spinning disk so I really don't get that argument.
If you have enough money to provision NVMe and still swap multiple gigabytes, just get a 16GB RAM stick for $40.
Laptops come with soldered RAM now where upgrades are not possible. And most people use laptops nowadays instead of desktops.
Buying another laptop to double your ram just because you hit the swap once in a blue moon is kinda wasteful.
What if you from time to time you only swap a couple of GB? Isn't that better than having your system completely lock up and need rebooting?
Why should I pay $40 when I have way more than 16GB of free SSD storage on my computer already?!
> it is still more than an order of magnitude slower than RAM, and is harmful for the disk cells.
I mean, if that's not an issue for you, be my guest. But that's not the global optimum.
Reusing hardware you already have clearly feels more optimal than buying new one because it makes things unnoticeably slower.
I treat my 1TB SSD like a loaner and after doing the math, based on my write patterns in the last 2 years, the SSD should wear out in about ..checks notes.. 12 years. I'm a lot more likely to replace it 2-4 years though to upgrade to a bigger and faster model anyway.
Plus, all storage dies eventually, that's why backups are important. You paid for it, you might as well make the most use of it while it lasts instead of trying to "hypermile it" since you won't be leaving it as inheritance to your grandchildren.
If you're wearing out your SSD life endurance in a few years, you have an issue.
This is true, but it still takes time. What kind of workload do you have where you swap out 32GB of RAM? If you are in this situation you almost certainly need to buy these additional 32GB of RAM.
As an example of such workload: a server, one of the many, that runs financial calculations from many web users. It's fine for it to become somewhat slower: we'll notice the performance degradation in the metrics and stop accepting new connections for a while. It is not fine for it to just die with OOM ― the users automatically will reconnect to other servers, leading to a sort of a thundering herd scenario.
Then you obviously have a workload where having swap make sense (and you could even add swap on the fly in such a case). But that is not typical at all.
I think swap is useful as a temporary memory overflow area, but that's it. Actually "running relying on swap" is harmful
Items 1 and 2 on the list sound like fiction. Item 4 sounds plausible (same for the Windows 9X series - sigh - where cache and swap would allegedly fight each other)
Swap is less critical now, and given that k8s has some pathological dislike of it (again I suspect based on vibes) I can see why people dont have it.
However when you are using close to 70% of your total ram, swap improves performance significantly. Also where I work, it also stop the OOM from killing my repo VFS layer when I'm compiling something.
> The core problem with swap is that nowadays you need gigabytes of swap to actually make a difference
It was always the case. but then a TB of disk isn't that expensive anymore.
Yes, but the author is most likely optimizing for a metric that I don't care so much in my per personal computer while dismissing metrics that I care about.
> It was always the case. but then a TB of disk isn't that expensive anymore.
But swapping out GB of RAM is expensive in terms of latency.
But it has to be GB to actually make a difference. If your swap is not full of GB of data, then obviously you don't need it (or it just becomes a reserve when you are in an OOM situation - but this is where you don't want to be with swap avaible either)
> But it has to be GB to actually make a difference. If your swap is not full of GB of data, then obviously you don't need it (or it just becomes a reserve when you are in an OOM situation - but this is where you don't want to be with swap avaible either)
I think the parent meant, that it barely needs to be read / write GBs at the same time. Not that that much swap is never being in use at the same time.
So you can use a lot of swap, but will never need to read it back in all at once.
I have stopped using swap on any computers, laptops, desktops or servers, more than 2 decades ago.
At that time, removing the swap was definitely a great improvement (with the condition of having installed enough real memory in the computers; but even with insufficient memory it was much better to have a few processes killed instead of having an unusable computer that had to be forcefully rebooted, losing everything that had not been already saved).
Perhaps the handling of swap has been improved meanwhile and in certain unusual niche environments having swap might provide some benefits.
Nevertheless, I doubt very much that this is true. I have never seen any such case where swap could have been useful.
When there is enough memory, there should be no writes to the swap, because such writes can only diminish the performance, and in an unpredictable way, because they can trigger the SSD garbage collector. If there are no writes to the swap, there is no reason for it to exist. If there isn't enough memory, the cheapest solution is to buy more memory, instead of trying to find a software workaround for that.
The memory pages that are not dirty can always be discarded by the operating system and read again later from the SSD if needed. No swap is needed for that. There exists no way in which swap can improve performance in comparison with having enough memory. Having to also discard dirty pages is just another way of saying that there isn't enough memory and you are willing to degrade the performance instead of paying for enough memory.
Swap memory is certainly never needed in personal computers, outside of temporary use in certain exceptional situations, when one would be willing to run very slowly a program that does not fit in the memory, due to lack of access to a computer with more memory.
Swap could be useful only when configured in a great number of servers that run a well characterized workload, where one could make a trade-off between application performance and total memory cost.
In a server environment you don't have the luxury of running your 4 GB needing app on a 128 GB machine - that means you have overprovissioned and are paying a stupid tax.
A modern server aims to run as close as possible to 100% CPU and RAM usage.
In those cases you need swap to kick out rarely used memory (initialization, ...)
There's really no way to tune a system's swap size for ideal responsiveness. You can have a 32GB RAM + 32GB swap system perform great, and a 32 GB + 8 GB system perform miserably.
The problem is the working set, which is an often hard to predict quantity dependent on runtime factors. You can swap out 30GB of unused junk to disk and have excellent performance, but if you're constantly going through 33 GB of data on a 32 GB system, performance is going to be awful.
So really want you want to avoid this last case from hanging the system is to measure the system's responsiveness and act in response to that, instead of going for trying to get rid of the swap file. You can do that with something like systemd-oomd, which starts killing stuff when things are starting to get slow rather than when some arbitrary usage number is reached.
Linux (long ago) always became unresponsive when running out of memory with swap. Removing the swap solved this problem and it remained perfectly responsive when running out of memory.
I have no longer used swap for many years, so I do not know the current behavior with swap, but I have seen out of memory cases on modern systems without swap and I have never seen again a case when the computer becomes unresponsive.
There is exactly one case where swap can be helpful: when
1. there are low traffic memory pages that can be swapped out in lieu of
2. page caches in the working set that would otherwise be dropped, and
3. you can't/don't want to pay for more memory.
In particular: swap is NOT for when your system is out of memory.
In an ideal world, swap would automagically move stuff that you are not likely use soon out of RAM even when you use poorly written programs that don’t do that explicitly. And, disk cache would automagically keep stuff in RAM that you are likely to need again soon even when using poorly written programs that don’t do that explicitly.
So, ideally, they work together to max the effectiveness of you RAM.
Imagine you want to write something down, so you ask me for a notebook. Now you start writing very slowly such that I can tell you're not going to need more than one page of that notebook for quite a while.
I don't want to take the notebook away from you because you're fussy and you insist on having the entire notebook to yourself, but someone else comes along also wanting to write something down.
Me being the clever kernel that I am, decide to let you keep a "virtual" copy of your notebook, but really only the first page of it is "real" paper and I give the notebook to the next person who ALSO insists on having an entire notebook to write down a few words on a page.
I can now have 20 people all writing in their own "virtual" 100 page notebooks, while I really only have a single 100 page notebook to go around.
Only when the combined total of all needed pages REALLY hits 100 pages will I start to run into issues.
Once you hit 100 actual pages of your notebook, I could either give up completely or I can say "Hey that first page you wrote to, I noticed you haven't used it in a while... I'm just going to store that in the filing cabinet."
Now again we're fine and we're even past 100 pages, but if you ever need to read the page I stored in in the filing cabinet, I'll have to swap that page out for another page, which will be time consuming and inefficient.
I mean no disrespect, but why in 20 years haven't you just read up on or watched a video explaining swap and disk cache?
Because that swapping would occur regardless, there would be a performance penalty when you didn't actually need it. Pointing the swap back to ram circumvented that.
I wouldn't say it's a crutch as there are still plenty of users machines and situations where ram might run out, so swapping is better than the apps crashing. But yes it's becoming increasingly obsolete.
The page file would have resided on HDDs, having the size fixed meant it would occupy one contiguous chunk of space on the platters thereby maximizing access speed and latency. A page file of variable size would inevitably become fragmented, which manifests as horrible access speed and latency.
Nowadays with the page file residing on SSDs this advice isn't relevant anymore, but that doesn't mean it wasn't relevant during its time.
As an aside, I still stick to the RAMx1.5=Swap rule because I'm just far too lazy to really optimize it, drive space is cheap now, and I still don't trust Windows to be smart with sizing it. Yes, that means I have a 96GB page file in my Windows 11 machine with 64GB of RAM; yes, it works and I don't care any further. :V
But for a lot of people (esp. in gaming but not limited to) fixed pagefile sizes caused problems when applications overallocated memory beyond the pagefile size (not necessarily used it, just allocating it was already enough to trip out of memory situation, I think).
because ram is expensive compared to disk.
Ram is normally fixed compared to disk
by storing not-used but still needed pages on disk, allows for much greater performance with less resources. for example it allows a bigger VFS cache.
If the OS is written so inefficiently that it performs worse without swap, put the swap on a ram drive to satisfy it without losing any performance.
- A user has a system with X RAM and Y swap. They buy Y RAM. If one compares (X RAM + Y swap) versus (X+Y RAM), it's obvious that they don't need swap anymore.
Thing is, the correct comparison is (X RAM + Y swap) versus (X+Y RAM + Z swap), where Z is a new amount of swap.