Conversely I've pondered using UDP packets as a "canary in the coal mine" for networks to monitor it's health.
Conversely I've pondered using UDP packets as a "canary in the coal mine" for networks to monitor it's health.
TCP (that is, most TCP flow control algorithms) specifically uses packet loss as an indicator of network congestion / how much bandwidth is available, and will back off and retry dropped packets at a lower rate; hence why TCP continues to function (albeit at degraded performance). UDP has no such mechanism, hence you see firsthand the dropped packets. A well-written application should back off in such a scenario so as not to flood the network, but of course many won't.
(Of course TCP may very well be prioritized over UDP! It's just not necessary to explain your observations.)
Obviously this presumes that the UDP protocol in question has some mechanism for handling lost packets. But, for instance, if you're doing lossy video or forward error correction, end users do not deal directly with lost packets.
It's a bad idea to measure UDP reliability on a quiet day and then make decisions based on those results. On a different day, everything will be happening at once - everyone trying to message their Mom, everyone trying to sell stock, everyone trying to cast a spell on the big monster - and that is when the most packets will be dropped.
You can use TCP as this canary as well by monitoring a counter of how many retransmissions have been performed.