Researchers discover major roadblock in alleviating network congestion
news.mit.edu
news.mit.edu
"We offer three key conclusions for CCA designers, who should model (or better estimate) non-congestive delays explicitly in delay-convergent CCAs. First, to utilize the link efficiently, a CCA must maintain a queue that is larger than the non-congestive delay on the path; second, this alone is not enough to avoid starvation, but in addition the variation in the queueing delay in steady state must also be greater than one-half of the delay jitter; and third, if we have a prior upper bound on sending rate, we may be able to avoid starvation while also reducing the queueing delay variation."
So what non-congestive delays are the problem? A key one is the way modern WiFi agregates multiple packets into AMPDUs for transmission to reduce the number of medium acquisitions it needs to perform. When WiFi is under heavy load this gives good throughput, at the expense of per-packet jitter. If I understand correctly, their conclusions mean delay-based congestion control loses the signal it needs to converge when the target queue size is similar to the non-congestive jitter. Many of us working on congestion control have wondered about this in the context of BBR, and it's great that this paper formalizes the issue.
The implication is that such delay-based schemes need to add perhaps a few tends of milliseconds to their queue delay targets to avoid potential starvation. Doesn't seem like a real roadblock, as per the article title, but the desire to reduce latency further does perhaps increase the incentive for more ECN/AQM research. It's perfectly possible to return ECN congestion marking before a queue starts to build, so this latency cost isn't fundamental.
Now, we (different company) have similar real-time algorithms, and the algorithms see much less problems (mainly across backbones of AWS, Azure, Oracle, IBM, Alibaba).
I suspect this is due to more bandwidth, more routes and better network optimization from the ISPs (we still see last mile issues but those are often a result of problems which better flow control algorithms usually can't completely solve).
Curious if ISP engineers can give a more expert view on the current state of the need or impact of better flow control in middle mile and/or last mile situations?
For public clouds who operate global networks, they can typically send video streams at low/mid priority all the way to "last mile" ISP by peering with so many networks and just running massive WANs internally. So they can get most of the benefits of prioritization, even though the internet doesn't support it.
In reality almost all large networks (internal and internet traffic handling) use MPLS[1] or some variant to tunnel/encapsulate different types of traffic and handle priority that way while not paying attention to whatever DSCP markings users can arbitrarily set. MPLS (in most cases) is invisible to the end user so the carrier can do their own QoS while not allowing customer configuration to impact it.
If "the internet" cared about DSCP you would definitely see the situation you're describing where everyone would just mark their traffic highest priority. Note you can still mark it, it's just that no one cares or respects it.
On your network and queues you can definitely use DSCP and 802.1p[2] (layer 2 - most commonly ethernet) to prioritize traffic. Thing here is you need equipment end to end (every router, switch, etc) that's capable of parsing these headers and adjusting queueing accordingly.
As if this isn't complicated enough, in the case of the typical edge connection (a circuit from an ISP) you don't have direct control of inbound traffic - when it gets to you is just when it gets to you.
Unless you use something like ifb[3], in which case you can kind of fake ingress queuing by way of wrapping it through another interface that effectively makes the traffic look like egress traffic. All you can really do here is introduce delay and or drop packets which for TCP traffic most commonly will trigger TCP congestion control, causing the transmitting side to back off because they'll think they're sending data too fast for your link.
UDP doesn't have congestion control but in practice that just means it's implemented higher in the stack. Protocols like QUIC, etc have their own congestion control implemented that in many cases can effectively behave like TCP. The difference here is the behavior in these scenarios is left dictated to the implementation as opposed to being at the mercy of the kernel/C lib/wherever else TCP is implemented.
Clear as mud, right?
Good news is many modern end user routers just kind of handle this with things like FQ-Codel, etc.
[0] - https://en.wikipedia.org/wiki/Differentiated_services
[1] - https://en.wikipedia.org/wiki/Multiprotocol_Label_Switching
The situation now is that bandwidth is plentiful. A lot changed in 20Y.
100G and 400G are now quite cheap - most of the COGS for a given router is the optics, not the chassis, control plane or NPU, and optics has been, on and off, a pretty competitive space.
Plus, almost all the traffic growth has been in cache-friendly content - non-video/audio/sw image growth has been modest and vastly outpaced by those. Not just cache friendly but layered cache friendly. Of the modern traffic types, only realtime audio-video like Zoom is both high volume and sensitive to latency and congestion. That's a small component, is often (but not always) either hosted or has a dedicated network of pops that do early handoff, and so on, so your typical backbone is now mostly carrying CDN cache misses..
> While Alizadeh and his co-authors weren’t able to find a traditional congestion control algorithm that could avoid starvation, there may be algorithms in a different class that could prevent this problem. Their analysis also suggests that changing how these algorithms work, so that they allow for larger variations in delay, could help prevent starvation in some network situations.
I thought some algorithms even do that already
not sure how this would work for tcp-ack's which can be cumulative. moreover, any approach that does this measurement must account for both delayed and lost/corrupt packets.
imho, only a true realtime jitter measurement would do, anything else would be a crude approximation, and might result in the same flaws as before...
[edit]: examples of '...anything else...' mentioned above might be inter-arrival histogram where receiver relies on packets being transmitted at a fixed cadence, in this case lost/corrupted packets would badly skew the computed numbers. another approach might be post-processing (after packet capture) where limited buffer space might prove to be the achilles heel etc.
Something mentioned elsewhere in the thread is wifi physical layers may wait to send small packets until they get a few small packets to aggregate or a timeout. Other systems that aggregate packets may do something similar.
On time division multiplexed systems, it may take measurable time to wait for an assigned slot, even if the channel isn't congested. Some packets would get lucky and have a short wait and others would have a longer wait.
This would be challenging to signal as the delay is added at a hop-by-hop level, but whether the delay is significant enough to signal is unknowable at that level. Maybe you could ask all hops to add to a field indicating milliseconds of delay (or some other fixed increments), but I don't think there's room for that in existing fields and you'd have a heck of a time getting meaningful support. ECN took how long to be usable on the internet, because some middleboxes would drop connections with it enabled, to say nothing of devices actually flagging congestion.
How would you know they're honestly reporting jitter?
Everything else will turn out to be a configuration burden, control plane load problem, security issue, hard to debug (especially after the fact, impossible) and so on.
Bandwidth and NPUs with designed-in minimal latency are easy to metric, easy to measure, easy to deploy, easy to implement and so on. They have very predictable behaviors.
Reality is that we are entering a phase where networks can be vastly simplified. MPLS is going, SR is here for the time being, QoS is dying, multicast dead, SDWAN is going to be a thing for a few more years then dead, and so on.