Hacker News Reader

Comment by devwastaken | Hacker News Reader

Anyone have recommendations on virtualizing windows for gaming? I'd like to keep my main desktop linux. Nvidia recently whitelisted direct gpu access, but I've been given conflicting advice as to esxi, virt manager, proxmox, etc. Surely this is a solved problem by now?

View on HN

I do this with QEMU/KVM with passthrough of an RTX 3090, an NVME SSD, and one of the onboard USB controllers. Works like a charm, though the VM boot time is very high if you allocate a lot of RAM to it (there's some kind of bottleneck in the linux kernel when pinning huge amounts of consecutive memory pages while using passthrough, don't fully understand it but it's a known problem).

Performance is indistinguishable from native, e.g. I can easily drive the screen at the max 144 Hz refresh rate, G-sync works, etc. I did put some effort in figuring out how to pin CPU threads to cores, optimize for the CPU core topology so Windows only gets cores on the same CCX (it's an AMD Zen 3, before that it was Zen 2), etc. But all of this is documented in many places.

Do note that depending on your motherboard not all of this is possible if the chipset & BIOS do not provide enough MMIO isolation, you might not be able to isolate a USB controller or a separate NVME drive, or you might get only half the PCIe lanes for the passthrough GPU if you need to use the other full-lane slot for something like a second GPU.

I would pay good money for a ready to use distribution/setup that supports this incl. making sure it continues to work after updates

The main problem would be to have to maintain the myriad of known working configurations, since they are all different depending on motherboard, BIOS, GPU, CPU, etc. If you are careful about picking the right parts (and assembling them properly), it's actually really easy to configure on any recent Linux distro, you can just click together the VM using virt-manager if you don't need anything special.

I know I had to jump through a lot of hoops to make it work though, my X470 motherboard didn't isolate the USB controller without a BIOS update for example, and after that the USB controller exhibited USB FLR (function level reset) problems causing it to hang the VM. This required blacklisting it's PCI ID from the Linux kernel and patching the kernel to disable FLR (fortunately these changes were later merged into the mainline kernel). I also had problems with the second GPU, if I plugged it into any slot other than the bottom x1 slot, the motherboard BIOS would reshuffle the IOMMU groups making it impossible to pass through the NVME and USB controller, or (if I put it in the second x16 slot) it would halve the PCIe bandwidth to the RTX3080.

All in all it took me the better part of a weekend to get everything working, but if I had to do it again from scratch and did some research into (in particular) the motherboard and BIOS, I would be able to set it everything up again in less than an hour.

I was so disappointed that there is no such thing available that its just plug-and-go. It really feels like this setup is breaking ground for some reason.

You need to have have a computer with integrated graphics / 2 graphics cards, and so can use a discrete card as a passthrough. Check out https://www.reddit.com/r/VFIO/

You don’t. Pass through to a VM on ProxMox works fine without integrated graphics and a single GPU. The graphics card simply drops the physical host console after the VM begins to boot, and then displays whatever the VM is outputting.

Do you have more info about this setup?

I used the method described here: https://elijahliedtke.medium.com/home-lab-guides-proxmox-6-p...

You're right, you can do passthrough with a single GPU, and there are multiple ways to do that. But note a huge caveat: you can't run an X server, whereas being able to simultaneously run desktop apps in the host and VM is a huge reason many run Windows in a VM rather than dual-boot into it.

Sure, it doesn't work for that use case, but most folks aren't going to be running a ProxMox system in a desktop use case anyway.

As of a few days ago, that info is outdated even for us consumers ;-) Technically one GPU will do, but it needs vGPU support to pass through slices of the GPU to the VM - which usually means expensive data center grade GPU (e.g. Tesla), because that feature is artificially limited by the driver.

However, there is some pretty awesome, cutting edge work done in that regard: Some great people very recently released a method for unlocking the vGPU stuff on consumer grade nVidia GPUs. When that was posted on HN[0], it only supported some 10xx GPUs; but with a merge from only 13h ago, most {9,10,20,30}xx should now work.

[0] https://news.ycombinator.com/item?id=26754351

Thank you for this information.

If you're okay with dropping vendor-unique features (e.g.: RTX) then I've found VMware's GPU virtualization to be very good, at least in my testing of Windows-on-Windows via VMware Workstation. Looks like Workstation 16 added DX11 support on Linux hosts, too.

Just make sure you go into fullscreen + (what they call) "exclusive mode", and maybe even increase the VM's priority when input is grabbed. That fixed any microstutters I had on my older CPU.

Although they're different use cases, I'm not sure if virtualizing would be the right choice for performance issues; I've successfully run audio+MIDI software under WINE using PlayOnLinux [0] which sort of [1] sandboxes each program into a virtual drive image therefore leaving untouched your ~/.wine directory, also allowing the use of different WINE versions for each program installed if needed. I also attempted to use virtualized Windows for MIDI software but the results were disappointing: although the software worked, it was unusable due to buffer underruns everywhere and huge latencies compared to either plain WINE or PlayOnLinux.

0: https://www.playonlinux.com/en/

1: It isolates the Windows program to a certain point, but I have no idea if it's technically a sandbox built for security which would protect the system against a Windows program willing to do any damage to or data exfiltration from the home directory.

I do this with ProxMox and a passed through AMD RX580 and it's working fine, but the disk performance is terrible, even worse than I'd expect for the hard drive it's on. Luckily that doesn't matter much for gaming.

The Proxmox wiki has great resources on GPU passthrough, but you may have to install a kernel mod for the AMD gpu not to take the host down when the guest powers off.

I haven't been able to do this with VMware ESXi (I've tried); I forgot if it worked in unRaid or not (But it should, in theory).

Is it possible to pass though a hard drive directly, like you do with the GPU?

Yes. If it's not used by the host at all.

Not with Proxmox, no. You can use a PCIe-Sata card and pass that through if you have the slots available.

I brought up a system with Xen and two video cards.

Though I would want to upgrade my hardware (ha, as if anyone can buy chips right now) before I made a setup like that my daily driver.

Take a look at the Linus Tech Tips YouTube channel. They have multiple videos about virtualizing Windows and passing GPUs to the VMs. IIRC they’ve used and benchmarked this setup for gaming and video editing. Quick search brought up the one for video editing[1]

[1] https://www.youtube.com/watch?v=1yFQd4MaKK0

in my very limited testing, QEMU/KVM had the least overhead.

Reply on news.ycombinator.com