Hacker News Reader

Comment by anticristi | Hacker News Reader

The biggest single cause of network engineer hair loss must be MTU and fragmentation. We had some customers over a tunnel (IIRC, it was MPLS) that reduced the MTU from 1500 (default Ethernet) to 1496. This happened while our vendor changed some equipment and forgot to update either MTU to accommodate the extra MPLS header. Of course, there was a misconfigured firewall that wouldn't fragment and wouldn't send "ICMP won't fragment".

The result? Most DNS queries went through. All chat applications worked. SSH generally worked, except when you started a full-screen terminal application. Smaller web pages load. Larger web pages loaded only partially.

Imagine non-technical users explaining their issue. "The Internet is half-broken. Please help."

God only knows how much hair I would have today if the world had figured MTU and fragmentation properly.

View on HN

> Of course, there was a misconfigured firewall that wouldn't fragment and wouldn't send "ICMP won't fragment".

Blocked or dropped ICMP has caused me heartburn as well. I am pretty sure my co-workers are used to my "blocking ICMP is Evil Bad and Wrong" rants by now.

AWS blocked ICMP by default though and cause a lot of traceroute problem for me...

> If the maximum transmission unit (MTU) between hosts in your subnets is different, or your instances communicate with peers over the internet, you must add the following network ACL rule, both inbound and outbound.

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network...

If your instances communicate with peers over the internet. So by default, AWS was built to assume your instances do not communicate over the internet, lol.

Some people don't want to deal with ping floods, sweeps, ICMP tunneling issued and the whole ICMP redirect attacks?

There's a difference between blanket blocking all ICMP and selectively blocking or rate-limiting a subset of ICMP messages according to an accurate threat model for your various networks. The latter, properly done, is unlikely to break MTU discovery or your own ability to troubleshoot and monitor your own hosts/networks. The former is what makes network engineers want to eviscerate you and use your gut as patch cables.

IMO, the crux of the issue is that kind-of-necessary and dangerous functionality is commingled in "ICMP", with no trivial way to separate the two.

With usual TCP and UDP, you block by default, allow outgoing, allow replies and your internet works (consumer defaults). If you want, you allow specific incoming ports and it's good enough for most use cases. It's almost trivial to configure a reasonable basic firewall, and you can learn all you need in 5-10 minutes.

With ICMP, there is this long list of types and subtypes, some of which sound dangerous while others necessary. Which ones should you block and which ones should you allow?

Do you expect everyone to create an accurate threat model? People have lives. So, some people end up blocking ICMP and internet "works" except .. of course it doesn't.

> Do you expect everyone to create an accurate threat model? People have lives. So, some people end up blocking ICMP and internet "works" except .. of course it doesn't.

This is the internet eqivalent of dumping your sewage in the lake because it's too difficult to deal with properly.

People that do this externalise the costs of their choices onto everyone else, and it takes a bunch of specialists to identify what's happening and educate the problematic network operator, and/or clean up the mess.

In a way yes, but I think the issue is largely caused by those up the chain that make it hard to deal with.

Doing a search on should I block ICMP, or what ICMP to block, answers are:

[1] No!!; some security issues; a lot of ICMP should be blocked; suggests further research

[2] you should selectively filter; example iptables rule to allow echo; assess evaluate and make your own rules

[3] listing of types and RFC recommendations for transit and local traffic

So I guess I should take "Should Be Dropped" and "Policy Should be Defined" from [3] and plug them into [2]. Why is it so hard?

Why isn't the answer: "No, defaults are safe, no need to block anything", or "Select one of: Endpoint / Site firewall / Internet router; customize if needed"?

IMO, this is a mess. This is the ultimate cause of all the sewage, time spent both debugging, and even more on learning what and how to block, configuring it all. Issues like this even hinder IPv6 adoption, because who is going to deal with all the complexity.

[1] http://shouldiblockicmp.com/

[2] https://blog.securityevaluators.com/icmp-the-good-the-bad-an...

[3] https://serverfault.com/questions/981558/which-ipv4-6-icmp-t...

> In a way yes, but I think the issue is largely caused by those up the chain that make it hard to deal with.

> Doing a search on ... answers are:

Sorry, "Google search results" are not "up the chain". That's your first mistake.

There's a tonne of misinformation in Google search results. It's real hard to identify what's good and what's not if you're not a specialist in the field, so it's easy to fall victim to this and just believe well meaning well written blog posts.

Don't do this. When you don't know something, speak to someone who does know about the subject matter at hand. Not anonymous people on the internet, but someone in reality that you can have a conversation with. If it's a topic that's vaguely within the remit of someone you work with, that's a good place to start.

I always saw ICMP blocking easy ability test for the firewall configuration person. If they block them this means they have no clue about the protocols they're configuring, and it's safe to assume that they likely got other things wrong too.

I consider the time I spent implementing RFC 4638 to allow the PPPoE implementation I use to have a 1500 byte MTU to be a good investment.

I wish centurylink fiber supported baby jump frames. :/

Would make more sense for them to just do IP over fiber directly than fiddling with their PPPoE implementation.

I have centurylink DSL with PPPoE and the thing that really bugs me is if your modem lost the PPPoE password, it can login with default credentials and ask for the passsword. CenturyLink clearly knows who I am without needing PPPoE authentication, so what do they get out of it?

My local provider requires PPPoE over fiber but accepts any password. It’s probably some legacy artefact that’s too complicated to remove from the stack.

Dollars to doughnuts the PPPoE implementation is tied into a billing system.

It makes wholesaling your network easy since you can tunnel switch based on the user@domain used during authentication.

baby jumbo frames.

haha. yeah, not sure how jump slipped in there.

The world did, then people came along and started blocking ICMP

Yes. There is a special circle in hell for braindead firewall admins filtering ICMP. Unfortunately happens often enough that "MTU problem" is the first thing that comes to mind at the grandparents description. Next in line would be "broken DNS"...

They are not braindead. ICMP used to be a bit broken and folks got sick of ping floods, sweeps, redirect attacks and tunneling hacks etc. Some pretty major places block ICMP as a result to keep their network reasoning and security analysis simpler.

Just because they are major places doesn't mean they are right or such practice buys any improvement in security worth the operational cost it imposes.

ICMP ping of death was a thing in the 1990s. Those bugs have been fixed for more than 2 decades now.

What do ping floods do that TCP syn floods don't? (I expect you run at least one service on the exposed hosts)

Also what system even accepts ICMP based route redirects by default from the open internet?

And when I am in my less charitable moods, I would claim that said people are unqualified to not just to configure networks, but to have opinions worth considering when it comes to setting network policy.

Unfortunately the only qualification required is for you to want to communicate with their network.

In my utopic world, we wouldn't need "ICMP won't fragment". IPv4/IPv6 would have the MTU of exactly 4096 bytes.

Why not a higher or lower level protocol? Because this is the lowest level protocol which is end-to-end and has a path MTU. Lower level protocols would need to somehow handle the MTU and higher level protocols would need to fit their segments within the MTU.

Why 4096? Because this is a common page size on computers. Multiple of this would likely bring little benefits, due to GRO/GSO.

Also, it's my utopic world, so if you don't like it create your own. :))

Not much of a utopia if all you do is say the same problem is now unsolved at a different layer ;). Nothing special about the layer below IP that allows it to fix the problem in a way IP couldn't be made to - it's just an annoying problem to fix efficiently.

4096 can be too much for unstable connections, for example Wi-Fi or mobile.

Also, 4096 bytes at what level? Layer 2? And when you are using a tunnel, you add bytes and thus have to go higher than 4096 or go lower and reduce MTU, or fragment the packet to keep 4096 bytes.

You are just raising the limit, not solving the problem.

This is the network version of ‘640k should be enough for anyone’.

Don’t fixate this kind of policy in such a fundamental protocol.

The issue is how the network are glued together, your X MTU will be embedded in a VPN tunnel, PPPOE - and that device needs to tell your device about its MTU

How would this work for IP over IP networks?

what is even more annoying is that there is a perfectly fine solution for this problem called MTU pad discovery. sadly many people block the entirety of ICMP to prevent this from working properly. Also, some NAT devices just straight up lie about their MTUPD size.

Websites half working should send alarm bells ringing that it’s an MTU problem. The reason google.com works is because they only support an MTU of 1280.

The minimum size dictated by IPv6, and thus almost always guaranteed to work.

I have an interesting uplink that drops packets of size about 1430-1470, but has no issue with smaller or bigger packets, even 1500.

ICMP is rate-limited, and 1500 byte packets work, so PMTUD doesn't kick. Hence, the connections hang if you hit the magic size.

Had to decrease MTU. Unfortunately, that doesn't solve remote-side UDP.

Why would you reduce the MTU? I had an issue many years ago playing league of legends where log in would fail until I changed the MTU to be slightly lower. Something like 1486.

Tunnels always have the danger of reducing the MTU because they have to account for the header.

Worse, is that if a tunnel doesn't reduce the MTU and its packets are dropped by a router further down the line the the ICMP TOO_FAT response back to the tunnel endpoint not the original host. The tunnel endpoint has zero clue what to do with it and drops it on the ground, leaving the original host in the dark.

Even worse is when overzealous firewall admins start locking down everything they don't understand and that includes ICMP. "Some hackers could ping our networks!" Then you're truly up a creek.

Luckily MTU breakage is fairly easy to spot once you know what to look for. It's easy enough to fix too if your local firewall admins aren't the block everything type. Just fix the MTU on the tunnel endpoint and suddenly everything will start working.

One final note: If as a router or application developer you ever find yourself having to fragment packets, please, for the love of all that is holy, fragment them into two roughly equal sized packets. Don't create one huge packet and one runt from the leftovers. It hurts me to see packets coming in from some heavily tunneled source with 1400 byte initial packet followed by 3 or 4 tiny fragments. Or worse, 3 or 4 tiny out of order fragments followed by the 1400 byte bulk because the MAC prioritized small packets in its queue and send them first.

> One final note: If as a router or application developer you ever find yourself having to fragment packets,

Ideally, you should not want to fragment IP packets. It is far better to do MSS clamping in TCP to prevent the size of the TCP payload to grew above the size of the max MTU in the path. TCP can stitch together the data at the other node just fine, without routers in between having to fragment the IP packets, which will kill your bandwith in comparison.

The "application" in this case is for example a packet->radio bridge application, or a VPN tunnel endpoint. Something that is chewing on packets mid stream where you don't get much choice in the matter.

One thing I like about IPv6 is the minimum MTU of 1280. That's big enough that if I'm really uncertain about the environment I can just set my MTU to that and avoid future headaches without impacting performance too badly. IPv4's 576 minimum nearly triples the number of packets you generate, which is really noticeable when you're running routers close to their limit. Forwarding speed is often dominated by packets per second, not bytes per second.

>IPv4's 576 minimum nearly triples the number of packets you generate

IPv4 minimum MTU is... 68 bytes.

>Every internet module must be able to forward a datagram of 68 octets without further fragmentation.

https://tools.ietf.org/html/rfc791

1500 bytes is the de facto standard maximum size of an ethernet frame (not including headers). Therefore, if you want to send data over Ethernet, for example, an IP packet you can typically only send up to 1500 bytes at a time, and if you try to send more it gets dropped. This is called the Maximum Transmission Unit (MTU).

Apparently, their customers were tunneling IP packets through another protocol, meaning that instead of sending an IP packet in a Ethernet frame, they were sending an IP packet in an X packet in an Ethernet frame. Since, like IP and Ethernet packets, the X packets need to contain some information related to the protocol, there was less room for the IP packet. I.e. the MTU was lower.

When you set the MTU on your Operating System (OS), it refers to something slightly different. Instead of "this is the maximum size packet that will fit", it means "assume this is the maximum size packet that will fit". You can use that setting to force your OS to send smaller packets if you know the MTU is lower than your OS thinks.

1500 payload bytes is not de facto, it's part of the actual 802.3 standard. Any other payload MTU is actually non-standard (and the IEEE has went out of their way to avoid standardizing other payload sizes. Frame size had a bump with 802.3as but still expressly left payload at 1500).

MTU doesn't refer to anything different in that case, MTU always refers to the maximum size frame "this" node can handle. It also doesn't always mean the OS assumes it's the maximum size packet that will fit on a path it's the maximum size Ethernet frame the OS knows will fit on the NIC. The OS has other methods for assuming things about a path. Forcing MTU lower does force the OS to assume any path is never more than the MTU though which is why it works as a fix.

MTU mismatches are hard to solve too. This happens when you have nodes and/or switches that are either misconfigured or don't all support autodiscovery and then have divergent defaults.

A typical symptom is that things (e.g., ssh) hang. This typically happens when you use a protocol that uses Kerberos in an Active Directory domain with very large Kerberos tickets. PQ crypto would do this too.

Reply on news.ycombinator.com