First thought was whitelisting USB vendor and device IDs, but I guess those could be spoofed. A button above every USB port?

Going back to PS/2 could be an option? Guess that wouldn't be too different from allowing all devices only on a single USB port.