Linux Performance: Why You Should Almost Always Add Swap Space

Swap engineering is definitely way more complicated than the article lets on.

What people often don't realise is that Linux distinguishes between dirty and clean memory, with dirty memory not reflected in swap, and clean memory already stored somewhere on disc, and program code is counted as clean memory that just happens to live somewhere other than swap.

Therefore, under memory pressure (especially if you set swappiness to zero), you will be preferentially swapping out your program code (because it is always clean) in preference to your program data. If you have no swap, then this is what causes the system to grind to a halt when the RAM is full- all your program code gets discarded from RAM, and nothing can run without reading it from disc again.

The recommendation to have a little bit of swap is absolutely fine. However, as the amount of RAM in your system increases, the penalty for running out of RAM increases as well. On larger systems (for example 256GB RAM), I recommend using something like EarlyOOM[1] to kill off tasks before the pathological swapping case occurs. Otherwise, you could end up with an unresponsive system. If you have lots of RAM, the kernel OOM killer waits far too late, and the system is already unresponsive.

[1] https://github.com/rfjakob/earlyoom

This is dangerous advice. At least on Ubuntu 16.04 with stock kernel running due to memory pressure into swap kills the systems. Basically everything times out and you can only hard-reboot. Without swap you have an OOM at least something you can recover from. If you need swap just add a little like 1-4gb - because if you have spinning disks 64gb swap won't help you at all because every request to swap is magnitudes more slower. Might be different with NVMe but it's still noticable.

In my experience, I think engineering swap usage is more complex than this article leads on. While adding more swap may make sense for many systems, I've personally worked on several systems where disabling swap made a huge difference. It's just one of those things where there isn't a hard rule.

One of the issues I've encountered more than once with Java software, is what appears to be longer than expected pauses in the GC, that I've linked to swap usage. The theory is, if you have a reference that lives long enough to get promoted to an old generation, but then is released, the memory in question won't be used for quite some time. This can lead the kernel to detect the memory pages not being used, swap them to disk, but then when it's time to do GC, the process is just frozen while the GC is running and get's backed up. Depending on the workload this might be fine, however, on both the platforms I encountered this issue on, were real time applications which were engineered to have available memory for the application and did (and also had no benefit from using the free'd memory for IO cache). So in this case, disabling swap had a real and measurable performance boost for us, getting rid of some of the variability in our GC runtime and pauses.

Under low memory conditions I think is also subjective. So if I'm a distributed / highly available system that's doing message routing in a telco environment, I would rather have the OOM killer kill the process, and failover to a backup, then for the system to be up but unresponsive because it's spending a majority of it's time swapping too/from disk. Again, this totally depends on the platform.

> As a last resort, the Kernel will deploy OOM killer to nuke high-memory process(es)

Yes! That is exactly what I want to happen!

When the system runs out of RAM, things will generally stop functioning, swap enabled or not. The only question is how you want it to stop functioning when that happens.

In almost every situation, I'll easily take the kernel killing whatever single process it thinks is most appropriate to get rid of, and keep everything else up and running smoothly, over grinding the the entire system to a halt by upping the effective memory access time by orders of magnitude.

Simply put: If everything doesn't fit in memory, then don't try to run everything!

Properly designed software nowadays is designed to be able to crash without corrupting data. As far as I'm concerned, it is almost always preferable to kill and restart instead of giving CPR to processes that don't fit in the working memory.

Kubernetes forces you to start it with flag to tell it "I know what I'm doing, I know this machine has swap, but start Kubernetes anyways even though it's normally a bad idea to run Kubernetes on a machine with swap".

I wondered this for a while, unfortunately a major question I have remains unanswered in the text: What about SSDs and SSD-only systems? (Like... any average laptop.) Should I be worried that putting a swap on an SSD will cause it to wear out fast due to many write cycles?

Cargo cult system administration.

No, you should not 'almost always add swap space'. What you should do instead is tune your system for its intended use.

If you need swap as an 'early warning system' that you're about to run out of memory you're already doing it wrong and the OOM killer is a piece of code that has default settings that can be tuned, ditto for the virtual memory manager in the kernel.

http://www.oracle.com/technetwork/articles/servers-storage-d...

https://access.redhat.com/documentation/en-us/red_hat_enterp...

Select appropriate equivalent for your kernel where applicable, and don't forget about the per-process limits in ulimit.

This sort of 'almost useless' advice is pretty annoying, it shows that the writer has an interest in the matter but didn't bother to dig in deep enough to make the advice truly useful.

Also, the comment on that article that implies that swap will allow you to update a running process because it is backing the image is wrong, the backing is simply done through the filesystem and swap space has nothing to do with it. The practical upshot of which is that you're not going to be able to reclaim the space held by the binary if you should remove it because the 'file is busy' until the last instance of that process exits. In the meantime, if you really want to run the newer version under the old name or path then you're free to unlink it or 'mv' it to a different location and to start the new binary from the default location. On some Unixes you can even overwrite the binary using an mv command making it look like the old one has disappeared but again, under the hood it will still be held open, you can check this using lsof or whatever is your local equivalent to see that the file is indeed still present. In a pinch (and you really really have messed up if you ever need this trick) you can re-link the running image by hard linking to the inode.

Interesting; Jeff Atwood of Coding Horror/Stackoverflow/Discourse forces swap creation as part of the installation of Discourse as a Docker container. He prefers slow performance to OOM. https://meta.discourse.org/t/create-a-swapfile-for-your-linu...

I've often disabled swap on systems running high-throughput databases like ElasticSearch and Cassandra because paging to disk will cause one or several nodes to slow down, which affects the performance of the whole cluster. The better thing to do in those cases is let the node fail right away by disabling swapping.

I fact, elasticsearch prefers to be run with `swapoff` : https://www.elastic.co/guide/en/elasticsearch/reference/curr...

When running an instance where the biggest process is a garbage collected runtime, one does not want any swap at all. If the garbage collector is forced to walk through all of the pages in swap hunting for live references, performance is terrible. It is better then to have the process see only as much virtual memory as there is RAM in the system. It is a failure of Unix/Linux that there is no sensible API for managing real RAM from the point of view of the process.

What do the people that use cloud instances that don't have local disks do? You definitely don't want swap to be on EBS...

Nowadays linux has this thing called zram disk. It compresses data in memory on the fly. I think general swapping advice would serve most people better if they always add swap on zram first, before swap on ssd or hdd. The difference is that getting data in and out of zram is so fast that you can actually use it as a tier of a slower ram, transparently compressed.

I'm not going to disagree with the article, because I haven't done any testing myself that would suggest that the point made are wrong. I do however find it amusing to login to a system with 32 or 64GB of memory with a 2GB swap. It seems humorous that those 2GB should somehow do anything if you already used 64GB of RAM.

System tuning is largely dependent on workload and use case. You can't just give a generic advice like this.

So, wouldn't the best answer clearly be to buy even more RAM, and use a RAMdisk for swap?

If you must use swap (you probably don't need to), then at least set zswap.enabled=1 in the kernel boot options, typically in grub. This will enable lzo compression of swap in memory so there is less writing to disk. Some newer kernels use lz4.

Either way, what most folks are actually missing is the correct kernel settings for the amount of memory they have. Sadly, the kernel does not dynamically adjust these based on your total amount of memory. Below is based loosely on Redhat suggestions. Please note, that if you fall below the min free, the kernel will decide what to do next, based on your oom and panic settings. The settings below will have the kernel free cache and other memory earlier so that you do not hit those stalling conditions and can even prevent some OOM race conditions. Some might suggest tuned, but use tuned with caution or at least read up on everything it does.

    MEM=`grep ^MemTotal /proc/meminfo | awk {'print $2'}`
    if   [ ${MEM} -gt 1129241478 ] ; then
        sysctl -q -w vm.min_free_kbytes=16384000
    elif [ ${MEM} -gt 564620739 ] ; then
        sysctl -q -w vm.min_free_kbytes=8192000
    elif [ ${MEM} -gt 352887962 ] ; then
        sysctl -q -w vm.min_free_kbytes=4096000
    elif [ ${MEM} -gt 176443981 ] ; then
        sysctl -q -w vm.min_free_kbytes=1024000
    elif [ ${MEM} -gt 88221990 ] ; then
        sysctl -q -w vm.min_free_kbytes=524288
    else
        sysctl -q -w vm.min_free_kbytes=262144
    fi

If you have small VM's, then perhaps set the default above to something a little smaller. You can of course free up about 128MB on default installations by removing "crashkernel" from your grub config and rebooting.

Then do this regardless, because overcommit set to 0 does not mean off, so we set the ration to 0 as well. Overcommit is good for developers testing code and finding the correct ways to manage memory in their applications during development.

    sysctl -q -w vm.overcommit_ratio=0

And of course, cache pressure plays into early evacuation of the right cache based on your usage:

    ## default is 100 (optimal for file servers).  4000+ for in memory databases.
    ## 10000 means always prefer page cache.
        sysctl -q -w vm.vfs_cache_pressure=1000

And if you have people oversubscribing a lot: (adjust based on your memory capacity)

    sysctl -q -w vm.admin_reserve_kbytes=131072
    sysctl -q -w vm.user_reserve_kbytes=262144

Please do read up on all of these before testing on your test machines. [1]

[1] https://www.kernel.org/doc/Documentation/sysctl/

Then finally, make sure you have Transparent Huge Pages disabled unless you know for sure you need it. THP can leak a lot of memory and it is nearly impossible to see without extensive kernel debugging.

In grub, set this and reboot:

    transparent_hugepage=madvise

Or to manually disable THP during run-time,

    echo -n "madvise" > /sys/kernel/mm/transparent_hugepage/enabled
    echo -n "never" > /sys/kernel/mm/transparent_hugepage/defrag

Then restart your applications. THP defrag can also cause stalling and lag spikes, especially in large memory java deployments (it will look like FGC's) and in MongoDB, Cassandra, others.

If you did this manually, stop your apps, flush cache, compact memory, then start your apps.

    sync;sync;sync
    echo 3 > /proc/sys/vm/drop_caches
    echo 1 > /proc/sys/vm/compact_memory

Some will say 3 sync's is not required. This is mostly true, but some old raid controllers treat this differently.

I always see people caring way to much about the amount of swap space used and not enough about the swap activity going on. It isn't the amount of swap space that slows a system down but actual swapping in/out of memory pages. In vmstat you want to pay attention to the "si" and "so" columns and in your alerting/graphing you want to keep track of the values "pswpin" and "pswpout" in /proc/vmstat. If these are almost always exactly or near zero then that means the fact that some memory pages are swapped out has virtually no impact on the performance of your system.

There are two other important issues to take into account though. 1) Even if the memory pages swapped out are not accessed generally they might be forces back into memory because of some specific action. One example is a database that has all the hot records in memory but other records that are accessed only very rarely swapped out to disk. In general everything will perform fine in this situation but the moment you do e.g. a table scan and a lot of these records need to be moved back into memory you might see a disk I/O peak that might be quite a kick on the neck for overall performance if the database is really busy.

2) If you don't have any swap space configured you might still run into problems with swap which seem to be caused by a bug in the memory handling in the kernel. I've seen this on some KVM hypervisors which were CentOS 7 systems. These systems were equipped with 128G of RAM and had two virtual machines running which both were configured with 32G of virtual RAM. They ran fine until one day the kswapd kernel process ran with 100% cpu usage even though no swap swap was configured whatsoever (to avoid the situation mentioned above). The "fix" was to dump the systems caches with "echo 3 > /proc/sys/vm/drop_caches" which seemed to calm down kswapd again. As best as I can tell what happened is that the system used all the free ram for the page cache and buffers and when the system needed some memory it apparently prefered to swap pages out to disk (even though no swap was configured) rather than reclaiming page cache of which there was plenty to reclaim. Unfortunately that means there seems to be no bulletproof way to say "only use physical ram and never try to swap anything out to disk". Even /proc/sys/vm/swappiness can be dangerous as a value of "0" doesn't actually tell the system to only swap if absolutely necessary but can lead to OOM situations even if swap space is still available (see https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux... for details).

TL;DR: 1) Don't just pay attention to the amount of swap space but to actual swap activity over time 2) Be aware of corner cases and bugs relating to swap

I'm sorry, but not a single reason was presented in favour of swap when you're 100% confident that you will not max out your RAM. Swap = completely useless slowdown if you have total control over your system.

Better use ulimit to prevent processes with e.g. memory leaks from causing havoc.

Swap isn’t needed at all in this age. It’s just there for people who can’t afford more RAM and therefore have to resort to hacky solutions like this.