Making Machines Move

It obviously is slower, but note that you take the hit only for the first time any block is read, and not for anything written after migration has started. Our migrations are overwhelmingly intra-region, as well, so really what you're doing approaching the performance envelope of the "standard" SAN topology used for cloud block devices anyways.

(I don't want to pretend to know what EBS does to make this fast!)

I still remember your paper about BigV! Very interesting solution and architecture. I was very close to becoming a customer essentially because of this. You were ahead of your time :)

I suspect you might want to have a quick look at the numbers in your first sentence and have a rethink ;-)

> I can't remember whether Linux supported the kind of fancy disc migration we wanted to do back in 2011.

I don't think it did. There was one other important difference (reading between the lines) between fly and us, and that's that we weren't using the kernel's NBD implementation, at all. At the time it had a hard limit of (from memory) 10 nbd mounts, so it was out of the running for the number of VMs we wanted on each head. We had to do it in user space, which meant we never had the problem of kernel threads locking up when the network had a moment. It was flexnbd talking directly to qemu, which presented a virtio block device to the guest.

That meant not using LVM2 or any of the other niceties directly in bigv itself. We did use it for administering the systems that ran bigv, and we used it in other places, but from memory the head and tail code itself had no knowledge of volumes or partitions. From the tail's point of view the customer disks were just files on a filesystem, and as far as I'm concerned that was a massive win - it meant that grabbing someone's VM to debug it needed no tools other than being able to recover the file, which isn't the case on some other storage systems.

It also meant that I ended up doing some of the most fun code I've ever written, getting the multithreaded code in flexnbd right. Although I do remember commenting fairly early on that threads in C might, perhaps, not be the finest choice of concurrency primitive for that particular job :-)

> It is particularly sweet trick to have a suspicion about a server and just say "flush it!" and in 12-24 hours, it's no longer in service.

Being able to do that with both the heads and the tails was nice, and it's fun seeing the same ideas we built crop up in other places. I'm not sure exactly how far ahead of the curve we were but it's definitely taken a while for some of these ideas to spread - largely, I suspect, because kubernetes came along and stomped all over the conceptual space, and it's just got different ideas about how a lot of this stuff works. I do know that no other nbd server at the time could do what we were doing correctly, because if it could have, we'd have used it.

That experience left me with a few lasting impressions:

1. Hardware RAID is a scam. Rebuilds take too long, more disks fail, the firmware is buggy, it's expensive. You're best off using the hardware as a convenient way to ram more disks into the chassis than using its RAID features.

2. You're lucky if you get 12 hours' notice. For storage volumes that size, where you're already on network storage, you want a snapshot to already be available off the machine when you make the decision to evacuate. And that means some synchronisation algorithm running all the time. There are systems around which do that now - Longhorn springs to mind - and that would have been the next thing to build if I'd convinced anyone it was worthwhile. I understand fly saying "Raft is too complex" but then the flip side is that you have to move all the data at the worst possible moment. Maybe the network speed to disk size ratios they're dealing with make it make sense?

3. Small teams. Small teams all the way. Fly are doing this with - depending on how you count - between two and four times as many hands on keyboards as we had (for an international product, where ours was in two DC's, so there's that). There's a break-even point for how much faster you go with more developers, and it's usually lower than ten. People often don't understand how fast a team that small can go because they've never seen it first-hand, but having since seen the other extreme... yeah. It might not have felt like it at the time, but we got a lot right in terms of enabling ourselves to move fast.

> I did miss this kind of work in 2022 and interviewed for a support role at fly. I'm not sure how to take being rejected at the screener stage, I'm sure some of my former staff might be able to explain it :)

No comment >:-) Having just spent a spectacularly unproductive afternoon trying to wrangle some particularly recalcitrant AWS terraform into a cooperative shape, I can say that this kind of work still needs doing, though...

> I'm not sure how to take being rejected at the screener

> stage, I'm sure some of my former staff might be able to explain it :)

Ouch, I'd have imagined you'd be more than qualified!

We don't use ZFS (in theory, we give Fly Machines a raw block device), but I'd be interested in the sketch of how you'd do something like this in an end-to-end ZFS system, just so I can see how well I communicated the properties of our system in this post.

The log-structured virtual disks (LSVD) mentioned at the end of the post seems to address this?

We did NBD in preference to everything else because NBD is so simple, and is the kernel's de facto answer to "abstract a block device away from the hardware". Shaun ran into reliability issues with it (which, for all I know, were just bugs fixed in more recent kernels) and iSCSI was within arm's reach, so it wound up being iSCSI.

We could do a bunch of testing and find the performance gains of NVMe-TCP worth the switch-out, and that project would be tenable (though: big enough we wouldn't do it unless the win was really big).

A serious NVMe-OF deployment would be tantamount to us building an EBS-style SAN scheme, which almost certainly would require us to single out specific regions to get the "better"/"newer" disk storage in, which is something we haven't had to do yet. I think in the next year or two we're more interested in seeing how LSVD, and especially Tigris LSVD, plays out.

No, you are correct. The database performance difference between using local NVMe and network-attached storage is integer factor on what is otherwise the same hardware. It would be surprising if this was not the case since storage bandwidth of local NVMe is typically integer factor higher and the performance of modern database engines is strongly correlated with available storage bandwidth for many workloads.

The network-attached storage does some things for you that local NVMe cannot, like easily moving storage between hosts, but it is also relatively expensive. Nonetheless, it can makes sense if performance and minimizing hardware footprint is not a major concern. If your main business is data-intensive on the other hand, the economics of network-attached storage start to make a lot less sense.

Most good servers use removable SSDs or HDDs, which can be replaced or moved from one server to another during the normal operation of the servers, without shutting them down. Even the consumer SATA connectors are designed for hot-plugging, which is also true for the enterprise NVMe or SAS connectors. Only the M.2 connectors are an exception among the modern SSD connectors, because they do not support hot-plugging, which is why they are not used in serious servers.

The only advantage of shared storage that is concentrated in some special place is that when such operations like replacement or moving are required, which require the presence of a human, the human may need to walk a much shorter distance to the rack with shared storage than if passing by each server had been required.

I fully agree with you, for what it's worth. I really like what Fly.io is doing here. Most of the advantages of being closer to the metal, combined to most advantages of being in the cloud.

This mentality is a rake we have stepped on repeatedly. Dragonboat in particular! It's wonderful, until you have to debug it. But even the most battle-tested and resilient Raft implementations create an immense infra/ops burden; a soundly built Raft implementation will eventually converge, but there's no guarantee it will do so within the tolerances of your SLOs without a lot of monitoring and continuous tuning.

Yeah, we need to tweak them; they were designed for a wider browser window than a lot of people use. If they're not rendered as actual sidenotes, I think we should make them footnotes or popups.

Migrating a VM by itself is simple.