When - not if - someone finds an implementation bug in either the browser or the USB hardware, we will see exploits far worse than keylogging passwords. The bug could be in any USB device, not just the subset you're thinking about. Do you really want to trust your host-controller, every USB device, and the browser interface to be bug free? Are you sure there are no subtle non-bug interactions between devices? Have you even seen how USB devices are designed?
Limitations like "no HID devices" are how it's supposed to work, which is different from how it will actually work.
edit:
Also, did everyone forget about BadUSB?