Did you decrease the USB ALSA period to the lowest practical value (eg. 128 or so)? In theory this provides finer-grained position updates and better stability at any given latency, but I'm not sure how it interacts with USB's polling rate.
Yes, the latency increases, the audio quality worsens, and I see the following messages in Pipewire's logs: