Latency -- You'll never get a wireless mouse faster than fiber-optic, since the computer screen would interfere with optical wireless transmission ;)
I jest. The real theoretical issue is pairing (portability). I can't think of a wireless solution that provides all three of:
1. No dongle. I don't want an extra thing to keep track of.
2. Fast and convenient pairing ("plug" and play). Connecting to a different computer should not be a hassle.
3. Secure. Device must only pair with computers of my choosing.
Dongles, WPS buttons, and bluetooth number confirmation are existing solutions that provide any two. A physical plug combines "connect" and "authorize" steps, neatly sidestepping the issue.
The closest thing I could think of to solve this would be a standard, as ubiquitous as USB, for wireless connection: each device has a code that serves as protocol negotiation, identification, and authorization. So you open up a "connect a device" dialog on the computer, type in your device's code, and the computer automatically discovers the device and pairs with it. But even this is a compromise on both usability (another thing to remember...) and/or security (it could be printed on the device, like a serial number, but then...).
And, of course there's the myriad practical issues mentioned elsewhere in the thread. I think if they were (truly) solved, I would buy wireless devices for my home setup, to do away with cable management, and keep cheap wired ones for use with other people's computers. I don't anticipate this happening for at least a decade or two.