Hacker News Reader

Comment by kibwen | Hacker News Reader

Can someone more knowledgeable explain why the digits 4 or 6 specifically would be causing issues?

Ethernet frames start with the destination mac address, IP payloads start with the version. It appears that in some networking equipment, there are places where it may have a ethernet frame or an IP playload and uses the first four bits to determine: if it starts with 4, it treats it as IPv4, and 6, IPv6.

So if it gets an ethernet frame destined for your address starting with 4 or 6, it will think it's actually an IP payload, and discard it because it doesn't validate as a IP packet or otherwise misdirect it.

Edit to add: In MPLS networks, 'P routers' (core routers) by design don't know if the payload is Ethernet or IP; but may want to peak into the payload to do hashed traffic balancing or validating packets before forwarding. Hashing the packet as an IP packet when it's an ethernet packet is problematic because you want all packets on a given TCP connection to have the same hash value -- but since you misidentified the type of packet, the bytes used as input to the hash aren't static for a given connection so different packets in the connection may take different paths and may arrive out of order.

Wow that is idiotic. How can you be skilled enough to program routers and still do something as obviously wrong as this?

In this case there simply isn't enough information in the packet to do the right thing so it's a question of choosing which wrong thing to do.

Seems like an attempt at being "clever". L2 equipment shouldn't care what kind of packet it is, but we're trying to subvert the original spec and come up with new hacks by "peeking" into the payload and make assumptions about what it is in order to rule or queue it differently.

Obviously in hindsight this is a bad idea and "idiotic", but I can't see the immediate fault of whomever first decided to try this a decade ago when MAC addresses were sequential, IPv4 was fairly abundant, and we didn't really even have smartphones and the explosion of connected devices.

Ethernet has always "cared" what type of upper payload its carrying. It has to, there are any other things besides IP that run directly on top of Ethernet frames. For instance 0x0800 means that the ethernet frame is carrying an IPv4 datagram.

Here is a list of all the ethertypes it cares about: http://www.networksorcery.com/enp/Protocol/802/ethertypes.ht...

Fire up wireshark and you will see these.

What the GP means is that layer 2 switches should forward traffic agnostic of what the Ethertype field says. That field only exists so layer 3 knows what it's dealing with.

I see. I misunderstood then.

This is MPLS, not Ethernet. No protocol field, only endpoints know what's inside.

You might enjoy watching https://www.youtube.com/watch?v=cXSwoKu9zOg&t=5s - a video recording about "What's Up With Poor Performance Towards MAC Addresses Starting With 4 or 6?" presented at the North American Network Operator Group (NANOG).

Infinite facepalm

An MPLS router would like to know if the encapsulated packet is IPv4 or IPv6. If it is, the packet can be safely put onto multiple links which have the same approximate costs. They look at the first byte because in IP that is where the version number is stored[1]. Because this assumption may not be true, MPLS can carry Ethernet traffic for example, some times non IP traffic can match. Additionally, until recently, there were no MAC addresses which started with 4 or 6.

[1] https://en.wikipedia.org/wiki/IPv4#Version

Yo dawg, I heard you like Ethernet, so I put an Ethernet frame into your Ethernet frame?

Where is this going? Who is trying to turn LAN into WAN and why?

[MPLS is] ... often referred to as a layer 2.5 protocol. It was designed to provide a unified data-carrying service for both circuit-based clients and packet-switching clients which provide a datagram service model. It can be used to carry many different kinds of traffic, including IP packets, as well as native ATM, SONET, and Ethernet frames.

https://en.wikipedia.org/wiki/Multiprotocol_Label_Switching

Sure, but what's the application for Ethernet in there?

I must come across as dense but Ethernet > MPLS > Ethernet feels like a "let's start over" moment.

VPLS is the standard use case for this scenario.

it's to offer normal ethernet service in buildings that are connected through some other kind of metro network.

in other words, it's cheaper than making your customer buy a bunch of weird proprietary network equipment to hook up their office wifi.

People want layer 2 connectivity between independent networks for a bunch of different reasons. Look into what's driving VXLAN adoption, for example.

Ethernet over IP is a thing too. Useful for bridging networks over a layer 3 link that you may be forced into for whatever reason.

The email references BCP 128:

https://tools.ietf.org/html/bcp128

"any application that defines an FEC that does not take measures to prevent the values 0x4 and 0x6 from occurring in the first nibble of the payload may be subject to IP ECMP and thus having their flows take multiple paths and arriving with considerable jitter and possibly out of order."

That's not enough context I feel for most readers. I'm quoting a bit more here which I think will better explain the issue:

"In the early days of MPLS, the payload was almost exclusively IP. Even today the overwhelming majority of carried traffic remains IP. Providers of MPLS equipment sought to continue this IP ECMP behavior. As shown above, it is not possible to know whether the payload of an MPLS packet is IP at every place where IP ECMP needs to be performed. Thus vendors have taken the liberty of guessing the payload. By inspecting the first nibble beyond the label stack, existing equipment infers that a packet is not IPv4 or IPv6 if the value of the nibble (where the IP version number would be found) is not 0x4 or 0x6 respectively. Most deployed LSRs will treat a packet whose first nibble is equal to 0x4 as if the payload were IPv4 for purposes of IP ECMP.

A consequence of this is that any application that defines an FEC that does not take measures to prevent the values 0x4 and 0x6 from occurring in the first nibble of the payload may be subject to IP ECMP and thus having their flows take multiple paths and arriving with considerable jitter and possibly out of order."

> "In the early days of MPLS, the payload was almost exclusively IP. Even today the overwhelming majority of carried traffic remains IP."

You can pretty much carry "Any Transport over MPLS" (AToM). As mentioned, IP is the most common protocol, of course, but you can "tunnel" almost anything you can think of over an MPLS path: Ethernet, ATM, PPP, HDLC, SONET, etc. Basically, you dump a frame into one side and it comes out the other unmodified -- regardless of the network in between.

As noted in the linked e-mail, an ISP providing (layer 2) transport has (little to) no control over the payload so there's nothing they can do to prevent running into this issue.

Reply on news.ycombinator.com