Hacker News Reader

Comment by lbhdc | Hacker News Reader

How does migrating a vm to another physical machine work?

VMware has been doing this for years, it's called vmotion and there is a lot of documentation about it if you are interested (eg https://www.thegeekpub.com/8407/how-vmotion-works/ )

Essential, memory state is copied to the new host, the VM is stunned for a millisecond and the cpu states is copied and resumed on the new host (you may see a dropped ping). All the networking and storage is virtual anyway so that is "moved" (it's not really moved) in the background.

The clever trick here is that they'll pre-copy most of the memory without bothering to do it consistently, but mark pages that the source had written to as "dirty". The network cutover is stop-the-world, but VMware doesn't copy the dirty pages during the stop. Instead, it simply treats them as "swapped to pagefile", where the pagefile is actually the source machine memory. When computation resumes at the target, the source is used to page memory back in on-demand. This allows very fast cutovers.

> VM is stunned for a millisecond

This conjures up hilarious mental imagery, thanks

You just bop it on the head, and move it to the new machine quickly. By the time the VM comes to it won't even realize that it is in a new home.

Up to 500ms per your source, depending on how much churn there is in the memory from the source system.

Very cool.

That is really interesting I didn't realize it was so fast. Thanks for the post I will give it a read!

This blog post is pretty old (2015) but gives a good introduction.

https://cloudplatform.googleblog.com/2015/03/Google-Compute-...

Thanks for sharing, I will give it a read!

They pause your VM, copy everything about its state over to the new machine, and quickly start the other instance. It's pretty clever. I think there are tricks you can play with machines that have large memory footprints to copy most of it before the pause, and only copy what has changed since then during the pause.

The disks are all on the network, so no need to move anything there.

In reality it sync the memory first to the other host and only pause the vm when the last state sync is small enough to be so quick the pause is barely measurable.

Indian jones and the register states

When its transferring the state to the target, how does it handle memory updates that are happening at that time? Is the programs execution paused at that point?

No, but the memory accesses have hooks that say "This memory was written". Then, program execution is paused, and just the sections of memory that were written are copied again.

This has memory performance characteristics - I ran a benchmark of memory read/write speed while this was happening once. It more than halved memory speed for the 30s or so it took from migration started to migration complete. The pause, too, was much longer.

Ahh I think that was the piece I was missing, thanks! I didn't realize there were hooks for tracking memory changes.

No, they keep track of dirty pages.

Stream the contents of ram from source to dest, pause the source, reprogram the network and copy and memory that changed since the initial stream, resume the dest, destroy the source, profit.

vsphere vmotion has been a thing for years lmao

Reply on news.ycombinator.com