It's a trap! Also, VirtualBox is a lot better than you remember it.
Does the majority even run something like vagrant linux image with a GUI and mouse?
There's ways to work around that though.. such as host the USB over SMB and then mount the share inside of the VM.. or use a cloud service of some kind.. but it's a bit more clunky.
USB passthrough seems like a basic feature of a hypervisor IMO.
That's kind of sad.