Was MPLS Traffic Engineering Worthwhile?

I always broke MPLS-TE into two categories - traffic engineering for bandwidth, and fast reroute (FRR).

Bandwidth engineering, like Bruce said, was religion for some people and anathema for others. It saved at least one company from bankruptcy, letting them run the network they had much hotter than IP would have.

FRR, though, was an absolute game-changer. Local protection to failure and sub-50ms (often sub-10ms) protection switching in an IP network? Immensely valuable. Link connectivity protection was easy to deploy and I will argue made a massive difference in an Internet network's ability to provide real-time services in the face of failure. Node protection and bandwidth protection? Less so. But MPLS-TE-based FRR was the first local protection mechanism that didn't require you to have a specific topology and which could cover 100% of all connectivity failures.

I'm surprised Bruce didn't call out this distinction, because they really are two separate things using the same underlying tech.

There is some extremely nice 2018 research paper on new routing paradigms that seems to be blowing out older stuff, and that is only possible because MPLS exists. The NSDI paper is linked below.

https://www.usenix.org/system/files/conference/nsdi18/nsdi18...

I guess it makes sense why the article says how it is central in Google's / Microsoft's network architecture to achieve performance gains.

Darn, I was really looking forward to a discussion of car culture and public transit in the Twin Cities. e.g. https://www.strongtowns.org/journal/tag/Minneapolis

Sounds like it's time for my periodic re-reading of "Netheads vs Bellheads" (https://www.wired.com/1996/10/atm-3/).

For Cisco? Yes! :-P

(Though, after 3 years of QA'ing the stuff, I gave up on MPLS and moved to other technologies. I was just not able to understand it to the deep level I wanted to.)

It's always a good time to re-read the Saltzer, Reed, and Clark End-to-End Arguments in System Design paper. "A too-real example" triggers a certain degree of paranoia.

Yes! However LDP and and especially RSVP weren't. Thankfully with Segment Routing and PCE it's kind of solved problem now.

Symmetrical paths with hard bandwidth control. I don't see how this can be achieved without on-path signaling like RSVP. Sure, in theory, we could do something similar with a PCEP centralized controller and SR extensions, but it is extremely difficult to tightly manage the bandwidth for all services from the controller. There will always be some inconsistency.

While we can argue that this is a niche use case, it would still be interesting to hear of any instances where this type of service is addressed using SR.

There is nother HN post on similar subject by Systems Approach [1]

Bruce also co-wrote one of the earliest books on MPLS with the original author of MPLS [2].

[1]Was MPLS Necessary?

https://news.ycombinator.com/item?id=32988417

[2]MPLS: Technology and Applications:

https://ieeexplore.ieee.org/document/900633

I had always assumed MPLS (and the even more hideous, SRv6) were designed by software engineers. If you are designing a hardware network processor, traversing a variable stack of labels at line-rate is pretty horrible.

Curious, is Karpathy in MPLS the same Karpathy in current AI tech?

As always, Bruce raises a good discussion here, but I’m disappointed in the lack of depth of this analysis. The article, to me, characterises this as a religious discussion, choosing between simple ECMP/multipath and MPLS-TE. I think this ignores the business reality of why one looks at deploying traffic engineering within a network, and the available approaches. I’m also a little disappointed that it characterises Google’s B4 and Microsoft’s SWAN as networks that rely on RSVP-TE (to my knowledge they do not, see the B4 paper). To my mind, these are demonstrations of the utility of traffic engineering independently of the mess that distributed traffic engineering with RSVP-TE creates.

(My background: I’ve inherited RSVP-TE deployments in a number of continent-wide, and global networks — which has involved driving standards to improve its scalability, and subsequently driving segment routing in the industry and production deployments.)

The issue one has at any kind of scale is that it is non-trivial to acquire capacity that can be coherent in terms of different optimisation dimensions for your network. For example, a network I worked on could acquire limited capacity on Europe to India cable systems, alternate routes were significantly different latency - but there was significant EU-India demand in the network. What were the options for placing this traffic on the network? IGP weights - sure - but this means there is no selective placement of that traffic (i.e., everyone has to take the same route), which one might not be able to support commercially. Looking beyond that, there are limited options _other_ than MPLS-TE based on RSVP-TE. Path Computation Engine (PCE) support, even when it emerged, was RSVP-TE-centric. So, commercially, those networks didn’t have a choice to deploy traffic engineering — and it wasn’t for want of trying. Significant cable system deployments have been driven on routes such as the one that I mention above — so there was capital to be deployed to fix the problem, it’s just that building such systems take years to be built. Did their architects want to deploy RSVP-TE? Pre-SDN (and SDN-in-the-WAN like B4 and SWAN) what option did they have to meet that business requirement? I would postulate very little (at least at the time that I was engaged in these discussions there were no clear alternatives). In fact, I would postulate that the existence of TE in B4 and SWAN shows that there is value in traffic engineering practically. Greenfield/ground-up systems still implemented.

RSVP-TE itself though was not well thought through. The systems design discussion that I think is very interesting here is considering the lessons that we can learn from such a technology. Distributed state in the network, that causes large amounts of signalling following failures, and requires midpoints to be aware of all demands that traverse them and admit them is fragile by its very nature. The scaling analysis that was done during the architectural work (RFC5439 for example) did not think of the RSVP-TE distributed system’s different points of dynamism — it concentrated on steady-state cost, but we’ve demonstrated time and again in production (over many, many years) that practically the system’s scaling was to do with the cost and scaling of dynamic resignalling following events rather than steady-state utilisation (I’ve presented effusively on this, see https://research.google/pubs/pub45800/ and https://youtu.be/NtED7CUHLNE).

Rather than raising the question, from a systems design perspective, as to whether MPLS-TE was a religious mistake — let’s raise the one around how complex distributed systems that have multiple vendors of their equipment (i.e., not ecosystems that are controlled by one party) can iterate on solving business problems without religiously filling in the gaps of protocols that don’t work. In my view, this would be a huge step forward for the networking industry.

Finally, let’s not fall into the trap that we over-generalise. Higher scale networks within a limited geography (terrestrial UK, US etc.) may not have the same business considerations — and therefore may not need the same approach to traffic placement. There’s, as always, a set of trade-offs here.