V3VEE – Open-Source Virtual Machine Monitor Framework for Modern Architectures

One of the take-aways from the project was that for a subset of workloads and environments it would be very nice if the underlying system supported large coarse grained resource allocations that could be managed independently. This isn't a new observation by any means, but its benefits are clear in a virtualized environment.

For instance with memory, one of the advantages Palacios had was that it preallocated a VM's memory at initialization using a set of large contiguous physical memory regions. This allowed it to represent the majority of the underlying memory map in a static and compact form that could be queried in constant time without locking. Nested page tables were managed independently and lock free on each core using local memory which, among other things, avoided NUMA crossings during page table walks.

Large physically contiguous memory allocations in Linux have never been well supported, and Palacios relied on memory hotplug to achieve it. This is certainly not a great way to do it, but there really wasn't another way to dynamically allocate large contiguous memory regions (hundreds of megabytes) at runtime.

Carrying that through to the hardware side, it would be nice to have an alternative to nested page tables that resembled something more like segmentation.