Hacker News Reader

Comment by hawski | Hacker News Reader

Seriously cool. That also reminds me of DragonFlyBSD's process checkpointing feature that offers suspend to disk. In Linux world there were many attempts, but AFAIK nothing simple and complete enough. To be fair I don't know if DF's implementation is that either.

https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpo...

https://www.dragonflybsd.org/cgi/web-man?command=checkpoint&...

View on HN

This article[0] on LWN seems to suggest that Linux has no kernel support for checkpoint/restore because it's hard to implement (no arguments there). But hypervisors support checkpoint/restore for virtual machines, e.g. ESXi VMotion and KVM live migration, so it seems like these technical problems are solvable. Indeed all the benefits of VM migration seem to also apply to process migration (load balancing, service uptime, etc).

0. https://lwn.net/Articles/293575/

IMO, big reason for why Linux doesn’t support it, is that big tech companies, who drive a lot of development and funding, found that it’s just not worth it.

Tech moved away from mainframe model of keeping single processes up for as long as possible and isolating software from the failures. Instead they embrace failure at the software layer, which gives you (when executed correctly) both same high level of availability, but also layers of protection against other issues (as node going down in expected way is the same, no matter what’s the root cause) and makes overall maintainability and upgradability systems higher.

Basically, why deal with complicated checkpoints at kernel level, that doesn’t understand your software, when you ca instead deal with that at the software layer, with full control over it.

Swap performance has been similarly underinvested.

How commonly do people need to use swap in the first place? On my current computer, I don't even have a swap partition because I've never gotten close to filling my 16GB, even with a big virtual machine and hundreds of browser tabs. In my experience on desktop systems, it just makes your computer painfully slow when you write a program that allocates far too much memory. Without swap, the OOM killer gets involved much earlier and you don't have to do a hard power cut to stop the disk from thrashing.

I just opened the site of the biggest retailer in my country, and looked at one entry-level laptop: it has 2GB of RAM. Not everybody has a high-end computer.

A Chromebook?

My $150 Chromebook has twice that; most computers these days tend to ship with more than than I think.

No, a normal laptop from a local brand.

When done well, swap can be pretty useful. I have a Fall 2019 MacBook Pro maxed out on CPU (8 cores) and RAM (64 GB) - MacOS still regularly compresses (highest I've seen so far: 12 GB) and/or swaps (highest I've seen so far: 16 GB) relatively unused memory to make more space for other applications and hot caches (which are still useful even with an PCIe SSD).

Keep in mind that everything feels fully snappy and responsive while this is all happening.

If you don't have enough memory for a single app, then your life is completely miserable - in that case, swapping is almost indistinguishable from crashed. But swap can be the difference between being able to run one app at a time, vs run three or four, as long as you only have one in the foreground at a time.

Swap doesn't play well with GC, or pointerful programming languages generally. Swap is a latency hit every time you touch an evicted page, and the more pointers your structures have, the more frequently that occurs.

In older times, when memory needed to be conserved, GC was less frequently used, and as a consequence of the accounting overheads of manual memory allocation, pointerful data structures weren't quite as common. Instead, you'd have pre-allocated arrays which supported up to N items of a particular type (and tough luck if you wanted more), or in more sophisticated applications, arena allocators. In such systems, the working set of memory is more likely to be contiguous; paging works better because every page load is more likely to bring in relevant data.

Switching between two open applications was the moment where you'd typically hit swap; that, and every now and then when you touched an area of the app you hadn't in a while, or jumped to a distant area of the document you were working on. The HDD light would come on, and you'd have to wait a few seconds while the computer ground through (HDDs being audible, some much louder than others) before you'd switch.

At some point in the mid 2000s, GC languages tipped the balance, especially JS. The browser became the main application, and wouldn't tolerate swap well. It became a mini OS unto itself, and had to balance the competing needs of different tabs. Then mobile exploded, and didn't use swap at all in order to compete with the iPhone, remarkable for its smooth animations. If there was to be latency, the application had to hide it; OS-level introduced latency on a page fault wasn't good enough, because if it happened on the UI thread, you dropped frames.

And of course swap never worked well on servers with frequent requests. Swapping on a server is a great way to back up all your queues.

What's hilarious to me, growing up in the era of swap, is that now you can have so much RAM that you can start caching your disk to RAM for a little extra performance (well, we've kind of moved past that, too, since SSDs are so much faster than spinny disks).

In fact, it's often worth moving some stuff in RAM to swap in order to have more space for disk cache (if you ever max out the RAM on a linux machine without swap you'll find it's a miserable experience because of the lack of cache and the fact that code from executables just gets memory mapped into the process address space).

By disabling swap, you are taking away your operating system's ability to trade less-frequently-used parts of memory for extra disk cache, which could hinder overall performance even if you have an excess of physical memory for your use case.

Honest question: what’s the difference (in terms of apps being killed) between 16 GB of RAM and 8 GB with an 8 GB swap?

There is no difference in terms of how much memory apps will be able to allocate before getting killed. But there will be a significant performance difference once apps start using more memory than the 8GB that is physically available (of course).

The problem is, system cannot know when you're going to use it. If application is swapped out and you use it after several hours, there would be excruciatingly long delay when it restores from spinning HDD disk. If I have reasonable amount of RAM I definitely prefer not to have swap.

Even if that means lower throughput from all your other apps over that several hour period? That is worth not having to wait a few extra seconds to go back to the less-used app?

Depends, but usually not. Harder to notice minimally better performance than to be annoyed by 2 second lag on application switch. ("this computer have 32GB RAM and it's trashing disk! grr")

>you don't have to do a hard power cut to stop the disk from thrashing

You can rescue from that situation by invoking OOM killer from SysRq magic key.

Yeah, and for exactly the same reasons.

In old days swap was necessity to keep system up, but nowadays you need it only if your dataset is really big (other than some weirdness in kernel, that actually like to have some small amount of swap to do cleanup and maintenance)

And if you have that problem, you have custom solutions that implement memory hierarchy for you, that give you much better result, as they can work across all the layers (cache/hdd/sdd/net/etc) in much more effective way than generic, software agnostic solutions focused just on one dimension.

They seem to be different problems though, as that LWN article suggests. It's probably easier to checkpoint/restore an entire image's state than an individual process' state.

But even calls like gettimeofday() work differently when running on hypervisors than they do when running on bare metal

The hypervisor migration features you mention are really powerful and stable. I founded a startup that does fast CI using them.

This is what it looks like: https://layerci.com/jobs/github/layerdemo/layer-gogs-demo/18

(in this run it restored a previous VM snapshot to skip the setup, and took a snapshot after the webserver started to give a lazy-loaded staging server)

There's also a project called CRIU that is used experimentally by Docker for container save/load: https://www.criu.org/Main_Page

You were faster than me. Yes, CRIU does checkpointing in userspace on Linux.

They contributed a lot of patches to the Linux kernel to support this feature. So there isn't any support for checkpointing in the mainline kernel. Rather, there are many kernel features that allows building such a feature in userspace. For instance, you can set the PID of the next program you spawn on Linux thanks to CRIU, because CRIU needs to be able to restore processes with the same PIDs as when they were checkpointed [1].

CRIU is used by OpenVZ, which, if I remember correctly, are moving or moved away from a kernel-based approach.

[1] http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using...

This was immensely informative, thank you for that link. CRIU looks like an awesome project. I can't believe I haven't heard of them prior to today.

CRIU and DMTCP are mentioned in the article.

Reply on news.ycombinator.com