This is also useful to disable modprobing obscure networking protocols just by an unprivileged process calling socket(SOCK_STREAM, AF_FULLOFBUGS, 0), which was a fruitful source of exploits at least a few years back.
However, for most server machines, if an attacker can plug a malicious USB device into your machine, something has already gone very wrong. (And for a good chunk of servers these days, they don't even have USB because they're VMs.) The primary threat model for this sort of thing seems to be laptops that are briefly left unattended and locked, or "Hey, can I charge my phone off your laptop."
For the unattended-and-locked case, perhaps you could block module autoloading until the user has logged back in. It's roughly similar to the approach Apple takes with Thunderbolt, I believe.
However, setting kernel.modprobe to /bin/false is extremely high-impact in reduction of attack surface compared to how much effort it is (one line in /etc/sysctl.d/); you can keep running a full-featured distribution kernel, even one from Fedora or Ubuntu that's simultaneously intended for desktop use.
There are a million easier ways to break into a virtual machine running on a physical machine you control than finding an exploit and writing code to emulate malicious hardware. You can just modify memory directly.