Hacker News Reader

Comment by amarshall | Hacker News Reader

Instead of using swap (on disk) at all, try deleting the swap partition and just use zram [1]. It essentially puts swap in-memory but compressed. This is likely faster than disk anyway, and avoids worrying about endurance at all. I don’t bother with swap partitions anymore.

[1]: https://www.kernel.org/doc/html/latest/admin-guide/blockdev/...

View on HN

Modern SSD are much faster at read than lz4, the fastest compression algorithm, at decompression and just 2-3 times slower in practice when writing than lz4 at compression. And they do not tax CPU while doing that, which is important since heavy memory usage may come from CPU-intensive tasks.

I did some benchmarks in my laptop and for heavy compilation tasks that may occasionally spike the memory usage over the amount of RAM memory compression was not beneficial compared with using SSD for swap.

The picture changes if one needs encrypted storage and cannot use hardware encryption on SSD. The software encryption will tax CPU on roughly same scale as lz4 and lz4 is still faster overall for read/write combination.

When I was using a Thinkpad X1 Carbon Gen6 with Linux it would hit a thermal limit and throttle the CPU speed if I read from NVMe at maximum speed for more than about 30 seconds. That was partly due to an issue with the way Linux and the BIOS interacted which gave a lower thermal limit than expected.

But it really depends on what NVMe devices you use, I've seen cheap ones bottleneck at 30MB/s when doing continuous writes with no downtime.

RAM is getting so cheap and easy to get that in recent years I only really use swap on laptops. Using a CPU core for compression isn't a big deal in a laptop because you usually have 4 cores and only 1 of them in use most of the time.

Huh? What numbers are you looking at?

On my laptop (i7-1165G7, DDR4 3200 MHz) lz4 decompression is ~3000 MB/s, compression is ~485 MB/s (on my 3960X desktop it’s ~1.5x that) (numbers from lzbench). Zram by default uses one stream per CPU, so theoretical max is (I think) 8x those for the laptop and, well, a lot more for the Threadripper. Non-Optane SSDs only really have a potential advantage in sequential writing. Max throughput aside, Zram will generally have better latency (and thus likely IOPS as well), and better consistency since most SSDs get their max performance from DRAM caches.

In the end, though, quibbling over synthetic numbers doesn’t matter much if you have benchmarked your actual use case. I don’t know what “not beneficial” means (the same? 10x worse?), but part of my original point was that

(somehow post got cut off, too late to edit)

…part of my original point was that Zram will resolve the SSD endurance problems.

With modern big SSD endurance is no longer the issue. You would need to write at max speed for many month before hitting the limit. I had a laptop with 512GB SSD that I used heavily for 3 years for development work and then for 2 years as a backup machine with light usage. Its write counter reports only 7% of the max usage. So on a new laptop with 2TB SSD after benchmarking things I just disabled zram/zswap. But then again, it was critical to enable hardware encryption on SSD rather than using LUKS on Linux. The latter is so slow [1] that zram is really a must.

[1] - https://www.reddit.com/r/Fedora/comments/rzvhyg/default_luks...

Well it all depends on how much writing one does. OP estimates they have 1 year before they use too much of their drive’s endurance.

Be aware that on-disk hardware encryption should generally not be trusted. It is not auditable or trivially verifiable. Some years back there were some revelations that some drive firmware encryption was woefully deficient. So much so that whereas Windows previously would treat on-disk encryption as sufficient, Microsoft stopped relying on it entirely.

I am aware of this. Plus there are constant power attacks when one disconnects the disk while the system is suspended when the key is unlocked on the disk.

But this is a development machine and I figured out that surprisingly non-trivial overhead of LUKS encryption in Linux is not worth it. On the plus side wake up from hibernation is very fast and to mitigate the risk of the above I configured the laptop to hibernate shortly after suspend.

You are right, formally lz4 is significantly faster than SSD, see https://www.reddit.com/r/Fedora/comments/mzun99/new_zram_tun... .

However, I did benchmark using stress-ng my laptop with Samsung 980 PRO SSD and when the stressors had to use swap, using zram or zswap was like 5%-15% slower than the real swap according to bogo ops counters that the tool reported. And the more CPU was taxed during the tests, the slower zram became while CPU utilization has not affected the real swap.

zram is a good thing. However in my investigations in 2014 "my conclusion was that swap wouldn’t be a problem as no normally operating systems that I run had swap using any significant fraction of total disk writes". During the last 8 years the amount of RAM in all my systems has increased significantly so swap is even less of an issue.

Sure but setting up swap is still recommended. I guess some internal linux memory algorithms prefer to have some safety net? Setting up a 1GB zram swap was effective for me, it‘s not much wasted memory as servers have so much memory these days and because of compression it can fit more than 1GB.

What I've seen is that under memory pressure kernel tasks trying to evict or free pages to satisfy an allocation request can race with other tasks dirtying and filling pages even faster, especially via the buffer cache. This can induce patterns of lock contention on low-level VM data structures and flushing procedures that effectively behave like a deadlock. Eventually various loop limits and lock timeouts will help unstick things, but in the worst cases the system gets caught in higher order loops and I've seen systems lock up for minutes. The systems I've seen this on never had any swap.

Some of the heuristics designed to minimize pathological contention latency seem to implicitly assume that the swapping subsystem--both in its ability to help free space, and the latency it introduces when loading and evicting pages--will help mitigate the chance tasks will get caught in a tight contention loop. IOW, the I/O latency of swap effectively induces back pressure on load, helping operations freeing pages to progress faster than operations consuming pages. (Predictably, the faster your swap, the less well this works. When people began putting swap on SSDs, heuristics had to be retuned.)

Arguably the root of the problem is the legacy of overcommit. Even though it can be nominally disabled, many aspects of the kernel were designed with the notion that the only direction to move under memory pressure is forward, relying on the promise of the OOM killer eventually freeing up enough memory to maintain forward progress on the current operation, rather than unwinding and returning a failure condition. The dynamic seems similar to buffer bloat.

Overcommit on POSIX systems is a consequence of fork/exec architecture of starting new processes. Fork must double the memory accounting if one disables overcommit and for a memory hog that starts a helper process that may lead to OOM when when in fact there are enough memory.

Windows, that does not have fork, does not suffer from this.

POSIX is not restricted to a fork/exec model, there's also posix_spawn(). Additionally it may be possible to somehow rely on COW to avoid double accounting after fork, provided that the new process execs "soon after". This would cover cases where posix_spawn() cannot be directly used, because some fixup needs to occur before the new process exec's.

Without double-accounting after the fork it is in general impossible to guarantee that no running program needs to be killed on OOM as the kernel does not know what the process is going to do after the fork. And fork still has legitimate users that does not involve exec. With multi-process architecture that many programs like browsers use for security it is a common optimization to have a seed process with sandbox initialized that is forked as necessary into a worker process.

A fast swap like zram or modern SSD is very beneficial on a development machine as one allows to keep active file cache for much longer so grep and git on huge trees works instantly even if one have a lot of memory hogs like VMs or language servers around.

Another potential offender is the hibernation process. It's probably compressed but still writing several GB to the SSD every time the machine goes into hibernation certainly doesn't help.

I am disappointed that in Linux the hibernation process is so dumb.

For example, if you hibernate a Linux machine, then boot it up and immediately re-hibernate, there should be barely any memory to write out (since very little will have changed since the last hibernation). Yet the written data is the whole image with no regards for the fact most of the data is already on disk and doesn't need to be rewritten.

The code and feature duplication between "zram" and "zswap/zcache" seems quite unfortunate. Is there no prospect of unifying these kernel-side tools in the future, so that a single configuration choice can be used to set up memory compression regardless of it being "backed" by a physical swap device?

Reply on news.ycombinator.com