Introduced latency on the capture side makes latency tuning your entire production pretty difficult. For non-real time usage, sure, latency in the 100-200ms range is more than acceptable (assuming it's deterministic, as you pointed out), but in the real-time world? Keeping things within a frame is pretty much required, and with the popularity of software-driven studio workflows across both amateur streaming and professional production, it's been real hard to get reliable performance out of USB hardware that didn't add frustrating amounts latency due to pre-ingest compression or seemingly random amounts of delay due to protocol or CPU time starvation.