I have a similar homelab setup and have been playing with USB over IP via https://www.virtualhere.com

It does require a client to talk to the server but seems to work pretty well. My thought is to setup a "thin USB client" on my desk that I can attach USB devices to which then are attached or switched to whatever metal or VM host I want in my lab.