Hacker News Reader

Comment by jeroenhd | Hacker News Reader

> Each operating system essentially gets a fixed allocation of RAM, something like 32-48 GB. This can lead to quite a lot of wasted resources, VM #3 may really need 64 GB instead of 48 GB at a point where VM #4 may have 24 GB to spare.

I don't know what virtualisation the author has in mind, but static RAM allocation isn't a requirement for VMs. VirtIO's memory balloon can be used to dynamically grow and shrink available memory. It's not exactly something you can do willy-nilly, but when a VM has a large amount of free RAM (like 24GB) you can definitely use ballooning to temporarily reassign memory capacity.

Since the author is talking about using Debian, they could opt for Proxmox to handle ballooning for them. Without Proxmox, scripts similar to this (https://github.com/berkerogluu/auto-ballooning-kvm/blob/mast...) could also be used to control memory distribution, though I imagine a server like this needs something a little more sophisticated.

View on HN

Since the author is running dual processors, they will need to account for NUMA domains. That is, half of the RAM is assigned to each CPU, and performance will suffer if a thread on CPU 0 is accessing memory belonging to CPU 1. numactl[0] can be used to bind a process to a specific NUMA node on bare metal, but a hypervisor can also set CPU/memory affinity for a given VM without fiddling about at the process level.

https://linux.die.net/man/8/numactl

Interesting. Would this still allow the virtualized machines to manage their own page tables as though native? I can't say I understand how it's implemented, but I'm concerned it might leave the host system with the same page thrashing the choice to use virtualization was intended to avoid.

You could examine LXC, which is a container setup, for isolation. Adding memory doesn't require a restart of the container. You may wish to examine ZFS or, going farther afield, even ZFS with FreeBSD with or without jails/bhyve.

They're trying to use VMs to run multiple page table managers on one single host. Containerization will use the same kernel's page management, so I don't think that'll solve the author's problem.

I don't know if this has any impact on the page tables, but the way it works is relatively simple: the hypervisor communicates with a driver inside the VM and tells it to "allocate" large amounts of memory. How much depends on the configuration; if you tell the hypervisor to configure a minimum of 16GB and a maximum of 48GB, then the driver will "reserve" 32GB of RAM. To the VM, it's as if some kind of driver is taking up 20GB of RAM, while I reality the hypervisor just doesn't allocate the RAM to the VM.

Then, when the VM runs out of memory, the virtio driver starts releasing pages for as long as the hypervisor lets it, slowly increasing the amount of RAM allocated to the virtual machine. This works until the hypervisor runs out of memory to allocate, after which the driver inside the VM will refuse to release pages and the VM's OOM will kick in.

The default is grow only (start with the minimum amount of RAM and only release more to the VM as time goes by), but it's possible to ask the driver to start "allocating" memory again with enhancements, like the ones proxmox do or by using scripting. With the guest tools installed, the hypervisor knows exactly how much free RAM each VM has.

This shouldn't cause problems for page tables on the client side as far as I know. However, certain workloads may not work well; if the VM allocates memory faster than the balloon driver canrrelease it, you'll get out of memory errors. Similarly, quick allocate-free cycles will prevent any auto shrink mechanism from working right.

You could try and run an experiment if you wish; the balloon driver works just as well regardless of whether you're allocating 16GB of RAM or 512MB. You could run a bunch of VMs with a model of the memory allocation you expect on your desktop and see if you end up page trashing like you fear.

I don't know how VirtIO does it but VMWare has a kernel driver which allocates the pages from the guest OS.

IIRC it locks these pages so they are unavailable until the driver releases them back.

that depends on your setup. ballooning is just a function to move memory from the guest to the host; if you move too much memory you'll get thrashing. the bigger problem is I/O indirection through the host; for I/O heavy workloads you'll want to use PCI passthrough for best performance, which (assuming it's working properly) should be identical to native performance.

Reply on news.ycombinator.com