Not gonna lie and pretend I know how it works but qemu is capable of passing USB devices into VMs while they are connected and visible to the host kernel, and this works perfectly for me for joysticks, steering wheels and other shit

For mouse and keyboard I just use the evdev forwarding thing where you press both ctrl keys to swap between host and guest

This is the biggest obstacle you're going to face:

https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

With AMD cards it's relatively easy if you're willing to install two cards and have one of them just sit there doing nothing when you're not using the VM. It's also possible to use just one card and detach it from the host system, pass it to the VM, and then reattach back to the host system when you're done playing, although I spent multiple days on this and never got it working. YMMV, it was 2-3 years ago, the driver support may have improved.

With nvidia it ranges from difficult to impossible.

The only good working solution I found (other than PCIe passthrough or specialized GPU virtualization) is VMware stuff (with better success on Windows [as a host]), because their DirectX virtualization is top-notch. Years ago I did some random testing of my Steam library and got close to native performance.

Obviously, it means you can't use DLSS, RT, or any other GPU-specific features, but their DirectX virtualization supports up to DX12.

external peripherals is probably the easy part, given that usb passthrough almost always "just works" in my experience. The bigger problem is getting GPU passthrough working.