It's not exactly about trusting the game. You'd be extending trust to... everything inside the VM you expose the graphics card to.

And graphics cards are messy. Many of them are frankly a lot like accessing main memory without an MMU. It's extremely easy to get scrapes of other application's video RAM that hasn't been zeroed.

I'm not trying to tell you you shouldn't do it of course. Just... be aware. "hijack the GPU" doesn't even require a whole lot of malice. I've had video memory of my firefox tabs from last shutdown draped across my screen while the lightdm login windows do their first paint, for example. This is just the world we live in :(