Passthrough-capable video cards aren't cheap or that easy to find and actually get working properly. It's 100% a nice dream, but no single major virtualization vendor supports it for desktop purposes intentionally. It's looking like for the foreseeable future it'll continue to be an expensive and royal pain in the behind.
Of course, YMMV, I recall seeing that linus derp tech guy YouTube make it work one time. Good luck keeping it working properly in the long haul.
I feel I've already learned this the hard way, have several spare Supermicro servers to show for the result of what I consider to be a failed project.
Instead, I maintain a separate windows-only machine for such occasions. Takes up very little extra space or time.
Would love to hear from other folks who use this on the daily.