Similar concerns should be made for Thunderbolt devices, which have direct PCIe access - much more low-level and dangerous than USB could be. The only system I've seen implement this is Gnome3 - it has a section in its system preferences for configuring Thunderbolt devices[0] and the Bolt daemon.[1]

[0] https://wiki.gnome.org/Design/Whiteboards/ThunderboltAccess

[1] https://www.phoronix.com/scan.php?page=news_item&px=Bolt-Pro...