AFAIK one could mitigate something like this by really restrictive udev rules only allowing certain usb drivers on certain usb ports (like no usb msc on the port dedicated for keyboard only).
If they're serious enough about finding 0 days and exploits to the USB or OS to load this shit any physical access to the box itself is off limits.
Consider what someone with the time, skill, and access to do that could also do without that: opening the case and directly installing some sort of device, planting a camera which records you typing passwords in (“oops, left my cellphone sitting out. Won't happen again!”), planting a radio receiver which opens up all sorts of side channel attacks, installing a passive network tap, etc.
A guard with a metal detector and strict limits on what you can bring into the building or what tools you can use inside is going to do a better job preventing all of those.
If a userland program can get at the raw HID interface, that can also be used for exfiltration to a tailored device.
xset led named 'Scroll Lock'