Create a QubesOS Gaming HVM with GPU PCI passthrough (2023) | Hacker News Reader

Create a QubesOS Gaming HVM with GPU PCI passthrough (2023)

forum.qubes-os.org

93 pointstranspute28 comments

2023, but has been updated and iterated on over time, so should be reasonably current.

It is... still an awful lot of work, though. I'm glad it can be done, though I've certainly not been motivated enough to try. I just keep a different machine around for anything gaming-ish, and don't worry about it.

That said, if you have skipped out on trying Qubes because you think you lack sufficient hardware, my Qubes daily driver (and primary computer for personal use) is a 2015-era Lenovo X250, with a 2C/4T I7-5600U, 16GB RAM, and a SATA SSD. It's limited in some ways, I can feel when I'm trying to do too much on it, but it is quite useful, and has been for some while. The memory balancing between AppVMs works nicely, and you can safely reduce memory allocation for a lot of VMs. A lot of my running Debian12 VMs are between 1GB and 2GB practical allocation, and work just fine.

Hi there. I just wanted to say that I like your blog, specifically the little kitty that follows your cursor. Incredible how endearing that little animation is!

Neko!

It's been software for many decades - https://en.wikipedia.org/wiki/Neko_(software)

I'm glad it's enjoyed. I very much like having him chasing around, even if it's a bit distracting at times.

Also worth mentioning Looking Glass which does a similar trick.

https://looking-glass.io/

I can't quite figure out what this is. Is it a way to run KVM VMs but make their windows show up on the host without a virtual "screen" that you have in a window?

My understanding, though I've not played with it, is that Looking Glass is a way to have hardware GPU accelerated guests "render in a window on the host." Normally, GPU passthrough for gaming has involved "You pass an entire GPU through to the guest, and use a separate monitor connection from that GPU for the output." So it's a VM, but not in the "interacts with the rest of the system as a first class window" sort of way.

I believe Looking Glass is a collection of techniques to pass a framebuffer from the GPU accelerated guest "monitor" back to the host, so you can have full hardware GPU acceleration in a guest, and still treat it like a window. I've not messed with it, though. Hm. Though I've got enough hardware to, right now...

What it does is uses a PCIe pass-through GPU to render the output, but instead of sending the actual video result to the actual GPU output (and thus, needing to switch input on your monitor), it renders from the GPU into a shared memory buffer which the host can display (in a window, or full screen). So it sortof works like remote desktop, but lower latency and full resolution due to using a shared memory buffer instead of TCP.

Yes. It's a way to have a "viewer" of the VMs screen that has as little latency as possible. My understanding is that this is achieved by configuring shared memory between host and guest. It uses some nvidia capture api/sdk to capture the screen of the guest and write it to that shared memory.

It has a server component running on the guest and a client component running on the host. I've been using it for a while with a windows 11 guest. It works but I wouldn't say it's production ready as I often get crashes. Though it's usually the server/windows component that crashes and not the client.

If going for a passthrough setup I'd generally recommend setting up evdev. It allows you to switch keayboard and mouse input to the vm by e.g. pressing both Ctrl keys. I then just have to change the video input on my display.

> change the video input on my display

Many monitors support input switching via DDC, using CLI software that can be mapped to a hotkey with keyboard macro automation, https://joostrijneveld.nl/posts/2024-06-04-ddc-input-switchi...

It's a good addition. The reason why I personally don't have this setup is ironically because of Looking Glass. Sometimes I want to switch my mouse and keyboard over but still want to just see the video output of my vm in a Looking Glass client window.

Does this work for GPUs without a physical display out port? For example an Nvidia A30 or H100

Looking Glass is for graphics remoting, including GPUs without physical display outputs.

It's not for GPGPU API remoting of GPUs designed for ML/AI.

Or moonlight for that mattwr

I appreciate that the journey might be the destination here, but you really don't need to do this anymore if the goal is to play games on Linux. Valve et al. have done so much Proton work that nearly every game I try just works.

Aside from the experience, this could be useful if you want to play games that have kernel level rootkits.

You still need GPU passthrough if you want to use ant actual GPU instead of llvmpipe or whatever else renderer. Mind you, QubeOS is virtualization OS, Linux on QubesOS does not have access to GPU by default

QubesOS is very much not "Linux on the hardware." It is silos of isolation, interacting freely at the window level.

I wrote on it some while back: https://www.sevarg.net/2023/07/29/qubes-os-silos-of-isolatio...

The concept is that you have a lot of isolated "AppVMs" running applications, in silos that cannot talk to each other. So, right now, I'm logged into HN with my "random-web" VM - that has nothing of interest in it beyond some PDFs I've downloaded. This, for instance, has zero access (except by exploiting and passing through Xen) my "sysadmin" VM, which contains my SSH keys for various things, and which I use for sysadmin type tasks. I've got other silos as well. All the windows from all these VMs interact like normal on my desktop - I'm not dealing with "Okay, this VM is for this, that VM is for that." Things just work smoothly and as expected.

A side effect of this, though, is that the AppVMs have no hardware acceleration of anything graphical. It's all software rendering in them. So gaming is normally right out - except, this talks about how to pass another GPU through to do this sort of thing, if you want.

> Things just work smoothly and as expected.

Do you have the ability to pass files between or drag and drop files between isolated levels? Can I mount a shared folder into multiple isolated apps so they all have access? I like QubeOS in theory but I’m hesitant to try it due to (perhaps misplaced) UX concerns.

If you _really_ want to break isolation between untrusted VMs, they should be stateless so that malware cannot persist after VM reboot.

"Converting untrusted PDFs into trusted ones: The Qubes Way (2013)", https://news.ycombinator.com/item?id=42401904.

You have the ability to push files from one AppVM to another - they appear in ~/QubesIncoming/[source-vm-name]/ and you can do what you want with them from there. I do this regularly and it works perfectly fine. It's not drag-and-drop, there be demons. It's either a command line utility or a context menu in the file explorer.

There's no concept of a local shared folder you can mount in multiple AppVMs, though as networking exists, you could easily do so, if you wanted - share a folder in StorageVM and mount it in others. Though doing so would defeat a lot of the point of isolation, if VM1 can corrupt a file that will then attack VM2 when read. You'd want to really think hard about your threat models and what you would and wouldn't want, with such a setup.

My advice? Dual boot. Yes, at some level, dual booting raises the risk of QubesOS compromise from the other booted OS, because it could pop /boot, and such. But in practical terms, you're no worse off doing that, than you are with another OS running full time - and as a lot of the threats are things like "ad-delivered malware" or "random PDF to your inbox being malicious," you gain quite a bit even if you still dual boot into something else on occasion. My daily driver X250 has a straight Ubuntu install on a separate SSD that I use on occasion, in theory. Mostly, it exists to run a VM to talk to a car, except I've not had to do that in a long while. Oh, and movie playback on long flights. Power efficiency for video playback on Qubes is "not good." Except I don't do those anymore either. So, every few months, I boot into the Ubuntu install and update it.

Caution: QubesOS, to the right sort of person, may as well be a virus. It took less than a year to go from "Play OS on a random laptop I was in the process of selling" to "Primary daily driver OS on several machines." And then I realized I didn't actually use the desktops anymore.

https://www.qubes-os.org/doc/how-to-copy-and-move-files/

This is exactly the architecture of Xbox. Every game is a VM

Qubes is not a GNU/Linux distribution: https://www.qubes-os.org/faq/#is-qubes-just-another-linux-di....

These rootkits hate VMs, and will almost always refuse to run inside one. In some cases, attempting this might get you banned.

I'd be curios to try vGpu with Qubes. It's definitely a security issue and has been left behind on newer NV consumer hardware but would be neat for low-risk qubes. I do have to admit that the performance is still great w/o hardware acceleration.

An old AMD workstation GPU supported SR-IOV partitioning, https://open-iov.org/index.php/GPU_Support#AMD

> It's definitely a security issue

Have there been public exploits of Intel or Nvidia SR-IOV implementations, to identify where hardening is needed?

Not that I’ve seen or heard of but my assumption is that it is mainly due to the relative rarity of partial/vgpu offerings outside of organizations. (Other than Blender render farming and some smaller upstart p2p gpu time reselling.)

I’d say Intel’s 12th gen igpu has “best supported” consumer vGpu offerings available and it’s through sr-ion I believe as well. I’ve had it on the back burner on my homelab until a recent performant gui need had come up.

A fee years ago (~2017-2018) I configured a machine with UnRAID where i had a vm running windows 10 for gaming. It worked great passed the hard configuration haha I was able to play on my windows 10 machine vm and work on a macos vm.

While it is possible and very cool to do PCI pass-through for GPUs, a huge problem with this approach is that online games have anti-cheat that checks to see if the game is running in a VM. So games without that mostly games that are single player will work but anything networked will likely not.

While it’s absolutely possible to play the cat and mouse game where you beat the Anti-cheat engine, It’s frankly an awful lot of work for such little benefit. You’re better off just using the service like GeForce Now Or something similar for networked games.

Reply on news.ycombinator.com