Hacker News Reader

Comment by viraptor | Hacker News Reader

> Compute node downloads such image, puts it on a local disk and boots a VM. This method makes it impossible to use the highly desired live migration

That's not true. Live migration is possible both with glance images and cinder volumes.

View on HN

Could you point out some docs about it?

It works by default. I don't think you need to do anything crazy about it. Base images will be downloaded by Nova during migration and libvirt/qemu will copy the differences. (assuming you use libvirt, I don't know about other supervisors)

See nova.virt.libvirt.driver:LibvirtDriver.pre_live_migrate - the code path for "if not is_shared_storage".

Are you sure that you are talking about true live migration? Is your instance available during the migration?

It's the openstack's "live migration". There's going to be a pause of course, but it's up to libvirt how long it's going to take. Reconnecting volumes will be always faster than copying the data.

You could also put the instances directory on a shared network drive - you don't have to wait for the copy then.

Well I am talking about true live migration where not a single icmp echo request is lost.

I still don't know which hypervisor are you using, but if it's anything libvirt-based, I don't believe such seamless migration is possible. Simply moving the networking information around is taking it's time. You're going to be down for at least the time it takes for example qemu to move the last bit of dirty memory over to the new host. Sending MAC updates to the routers will also take some time. Since ICMP is not retried, it's going to be lost.

I am using libvirt with KVM. I have successfully migrated many instances without losing any icmp. I understand that memory must be transferred. It just works fast in my environment. MAC address does not change if I remember correctly.

You simply avoided the race condition. It won't happen every time and it doesn't mean that the migration would never lose any icmp packet. It's impossible without organising the whole network around this idea.

MAC doesn't have to change, but the upstream router still needs to be notified about the new location of your instance. That's why nova sends the gratuitous arp packet (more than one actually since they are not guaranteed). At least that's the case for the old networking model. Not sure what's needed for quantum updates, but that will require at least one RabbitMQ message on Nova's side and an HTTP request to Quantum.

Another way to explain the problem: you can write to local memory faster than you can move those changes to another host over the network. Any application heavily processing the data and using all available memory will make memory pages dirty faster than it takes qemu to move them across to the next host. The result is that your instance has to be paused until the copying process completes (unless you're running on a non-standard architecture). Otherwise the copy would never complete.

What technology short of hardware FT (which I've worked on BTW) will prevent that? You'll lose plenty of ICMP echo requests before you even realize a migration is necessary, and no storage infrastructure is going to help with that. Why single out one when it's a problem common to all?

This is definitely possible, it is called block migration. Here is an example of someone showing usage (note the bug he mentioned has been fixed):

http://www.sebastien-han.fr/blog/2012/07/12/openstack-block-...

Note that there are some reliability issues using versions prior to qemu-1.4 and libvirt 1.0.2

To enable "true" block migration where the server remains live the whole time instead of being paused, you need to modify a config option:

block_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_NON_SHARED_INC,VIR_MIGRATE_LIVE

(this adds VIR_MIGRATE_LIVE to the default flags)

Also, keep in mind the same caveats to regular live migration with this flag in that there are edge cases where the i/o in the guest is so great that the migration will never complete.

You are right. Block migration is definitely the way to go. Unfortunately it is poorly described in openstack docs for KVM. http://docs.openstack.org/trunk/openstack-compute/admin/cont...

Reply on news.ycombinator.com