If you want low latency you really need to use USB3/USB-c. USB3 doesn't use polling like USB2. USB2 latency is measured in milliseconds. USB3 latency is measured in microseconds.
I'd optimize lots of other things like trying to avoid the compositor, higher refresh rate monitors and lower travel keys before I'd look at USB3.