Did you decrease the USB ALSA period to the lowest practical value (eg. 128 or so)? In theory this provides finer-grained position updates and better stability at any given latency, but I'm not sure how it interacts with USB's polling rate.
[W][30498.719355] spa.alsa | [ alsa-pcm.c: 2478 spa_alsa_read()] steinberg_ur44_mono_in:UR44,0,2: follower delay:4042 target:4832 thr:256, resync (374 missed)