“Gil Tenne - How not to measure latency” some of the best infotainment on YouTube. Sorry no time to find exact link. There are many.
Of course there be dragons. Oversubscription in various parts of the network can have interesting behaviour. Especially when coordinated user behaviour or complicated packet processing is at play. Think encapsulation or deep packet inspection. My worst case experience includes Cisco Nexus M1 32x10Gbps line cards maxing out at 1-2 Gbps throughput and 1+ sec worst case latency because of OTP. This is a datacenter core switch. And F5 WAF eating random packets because something looks like a VISA card number, causing a retransmit, that again shows up as high latency at higher levels.
With best case latency you can determine if the service can be suitable for real-time or not. There will always be buffer effects and they will vary by other user activity and the complexity or the packet processing. These effects will largely be unknowable in advance and the part that is knowable is extremely difficult to communicate to an average user. They don’t really get HDR histograms https://github.com/HdrHistogram/HdrHistogram.NET/blob/master...
ALL NETWORKS have perceptible jitter due to this, unless you rigorously apply FQ and AQM techniques to each slower hop on the path
Fatter tests like waveform show the common bufferbloat scenario in ways humans can see, better. But slow start overshoot is always there, on any connection that lasts long enough, which only takes a couple RTTs. You can clearly see netflix doing you in here, for example...
https://www.youtube.com/@trendaltoews7143
(except that in this case, the link has libreqos and FQ on it, so all that buffering is just local to the netflix flow, invisible to everything else)
Joking aside, the https://www.waveform.com/tools/bufferbloat test looks to see if the networking software is working correctly by putting a large load on the network, and then seeing if other streams are affectec by the overload.
The example on the https://libreqos.io/ home page is of * good software delivering 9 and 23 milliseconds down/up latency at full load * bad software delivering 106 and 517 milleseconds latency under load.
It is, in effect, a test for software failure under load
The M1 cards I mentioned was notorious with 1GB (?) buffer and a massively oversubscribed packet processor. Causing extreme jitter.
too many people in his world think tcp is a function call.