I guess what I'm picturing is the first time you plug in your keyboard/Yubikey/whatever, your system prompts you "A new device 'AcmeCo USB Keyboard' (VID=xxxx/PID=xxxx) wants to provide service(s): Keyboard Input. Allow? Once/Always/No/Never." After that, your system saves that combination, and assuming you authorized it permanently, whenever you plug in that VID/PID, it has access to be a keyboard.
If it tries to present another HID later, that would result in a new prompt, the same as a brand new device. If it changes HID while plugged in, that could even be a more severe message along the lines of "Hey, this keyboard just tried to become a video adapter and thumb drive, that's really suspicious and you should probably destroy it with fire".
Likewise, VID/PID changing without being unplugged/replugged in, changing too quickly, or even just after too many new VID/PIDs in a short time period should result in a temporary lockdown -- in the same way fail2ban blocks repeated login attempts from a given IP.