Yeah, this has been my experience as well. Even cheap Firewire audio interfaces are light-years better than their USB counterparts.

I still use a USB 2.0 MIDI controller and the latency is absolutely killer. Press a key, a beat later, see the note appear on piano roll...