Hacker News Reader

Comment by londons_explore | Hacker News Reader

This API is only used by one project to my knowledge...

CRIU is a project to save an application or containers complete running state to a file, and then restore it elsewhere.

It comes with lots of caveats.

I use it a lot (without criu or libsoccr) for high-availability shenanigans, to avoid the reconnection delay. Machine A is 'main' and had established a connection to distant-machine 1. A crashes (the hardware stops), machine B takes over its MAC, IP and all its established TCP connections. Nothing can be seen from the distant-machine side (maybe a gratuitous ARP slips out... to no avail). And yes, for <1 millisecond takeover this is necessary and I thank the criu project with all my engineering heart for not going the 'just put a module in there and be done with it' but actually making sockets checkpointable and restartable, and saving me from the pains of a userland network stack.

The actual details are far more funny and interesting (we could talk about checkpoint not being an atomic operation for the kernel, how you need to do some magic with "plug" qdiscs and qdiscs being applicable on egress only you'll look into IFBs and I love Linux it is so versatile and full of amazing little features). Don't forget to hot-update conntrack too...

And since libsoccr is GPL you might need to do this yourself, and you'll want to do it anyway, because it's interesting and you'll learn so many things.

My only gripe is the checkpoint still being a bit slow and maybe if I keep annoying Jens Axboe on twitter maybe soon it'll be a io_uring chain <3.

If you don't mind me asking, what kind of workload requires this kind of "high-availability shenanigans". Sounds fascinating.

9-1-1 callcenters often do, some use specialized server hardware that can in <1ms switch from one motherboard/CPU to another when it fails.

Apparently there are some workloads in finance that use similar hardware.

> Apparently there are some workloads in finance that use similar hardware.

Not surprising, I've heard from someone who worked at a financial institution doing high-speed trading that basically every ms counts for them.

Every microsecond, and HFT people rent datacenter space the closest possible to physical exchanges... When speed of light is your main concern, maybe the Linux kernel is not your friend anymore (although dpdk can help here). I'm happy people keep pushing the kernel so hard, and still try to keep the kernel generic and composable, so we can profit from this huge, amazing work.

Often times it's mission critical systems where latency is key, for many reasons:

- you have a human in the loop to take a split second decision, every millisecond counts

- you have a very short time to perform 'looped' operations - where the result of one measurement must be taken into account to effect the next measurement (adaptive optics, some radar systems, some mechanical control loops) and you can't wait.

You'd think 'oh but you got more than 1 ms for that' Well not always since one must take into account the time to detect the failure, and the time to switch other parts of the system (which sometimes must be done in sequence with the connection takeover).

I'd say 'forget tcp' there but we don't always get to decide the comm layer...

If RDS could fail over in 1ms I would be extremely happy. In practice, a few minutes of downtime for things like DB upgrades is usually acceptable in the business I am in, however this is enough time to cause quite a lot of alerting noise.

If the window of unavailability was instead 1ms, there would be dramatically less noise, potentially none.

Such redundance shenanigans used in Cellular networks, single node could handle thousands calls at the same time.

How do you know about the other machine's outgoing connections in real-time?

You capture and stream the state of the connection as often and fast as you can. Ideally the tcp state would be streamed in the same operation as checkpointing, still in io_uring.

CRIU is a really impressive tool, which can do some really cool things. It's pretty difficult to use, given the nature of what it's doing and all the moving parts to orchestrate. I added experimental support to Docker for migrating containers, but for lots of reasons it never made it as a full fledged feature.

Since the ELF interpreter and glibc initialize differently depending on cpu flags, it's probably unlikely that migrating between arbitrary hosts can be expected to always work.

https://github.com/twosigma/libvirtcpuid works around this.

(But in practice, we mostly rely on the fact that if you are in Google Cloud and specify a minimum CPU version, Google masks CPUID in that VM to exactly match that CPU, even if the underlying hardware is newer. They do this so live-migration of the whole VM works, but it also helps for process migration. I'm not sure where this is specifically documented but see e.g. https://stackoverflow.com/a/44507857 .)

Probably that's why hyperv have a option to force any guest only able to access to a common cpu instrument set. So they will always have the same flag and migration is possible.

Reply on news.ycombinator.com