Comment by toast0 | Hacker News Reader

Comment by toast0 | Hacker News Reader

Parent Full thread

It probably depends on who's running the DC, and how well traffic is separated.

Path MTU detection is all sorts of fragile, so exposing a tcp maximum segment size implying a MTU above 1500 is asking for trouble; actually even just 1500 isn't always a good idea [1]. There are many networks where too big packets are dropped without notifications, and if the other end is also on a 9000 MTU LAN, then you can get stuck. sigh

On the host side, packetization is not that big of a deal; nics can help, but it's also just not that many packets. Where hosts tend to run out of cpu is when you're dealing with much smaller packets.

Larger packets could help routers of course. And larger packets would probably mean fewer acks, which would be nice too. But, it's unlikely to happen unless mtu probing is enabled in more places.

[1] The best is to send the lesser of (theirs - 28) or your actual value. Well best is relative, this will result in the most successful connections, at the expense of more packets for networks where people aren't insane.

The main reason why Path MTU detection is fragile is that many people ignorantly filter all ICMP traffic (including type 3). But educating people about ICMP and its role in PMTUD [1] seems like a lost battle. I wish to be proven wrong but I have an impressions that younger developers, devops/SRE and even network engineers know about network protocols less in than 10-20 years ago. Nowadays I rarely meet people how know what Path MTU Discovery is. I even stopped asking about it on job interviews (for devs/ops) because no one can answer, it's a luck if a candidate can tell anything about ICMP at all.

[1] http://www.znep.com/~marcs/mtu/

The two things that boil my blood when dealing with SecOps people are:

1) Dropping all ICMP packets on the floor for "security" reasons, which means that basic diagnostics is now impossible, you get random issues with software that uses PING, and Path MTU Discovery is broken forever.

2) Leaving internal firewall ports with the default settings, which are intended for Internet-facing ports. For example, for "denied" incoming traffic from the Internet, the correct response is to silently drop the packet. Internally, the correct thing to do is to respond with a NACK to instantly close the connection. Without this, you spend days and days chasing down random and difficult-to-troubleshoot timeouts and weird 30-second delays all over the place.

As a random example, Windows RDP makes a HTTP call out to the Internet from the server to verify a CRL. This is safe and secure. Without this, bad certificates can't be blocked. Unfortunately, this happens in the "SYSTEM" context, which tends not get proxy settings applied to it, so it is often blocked, with a 30-second timeout. This failure is cached for 24 hours on the server, which causes a maddening delay when you connect to servers. Every day. Every server. But you can't reproduce it, because the second time it won't happen. (It also won't happen if anyone else connects to the server right before you.)

Years later, I still get angry remembering the snarky comments by the firewall guy saying that I'm just imagining things.

That's one reason, but there are several other common issues.

At high volume mtu bottlenecks, the routers dropping packets are likely to limit how many ICMPs they send; otherwise they'll run out of CPU. This wouldn't be terrible if ICMPs weren't so commonly dropped.

Some bottlenecked routers don't have globally routable IP addresses, and may not be able to send ICMP at all. This can happen inside long haul networks, where there may be tunneling that reduces the effective MTU. This seems less common today, but I dunno? One 'recent' advance is that some PPPoE networks use 'mini jumbo' frames, with an ethernet MTU of 1508, so that the PPP mtu is 1500; that may be happening on other layers with tunnels too.

Yes, unfortunately it is not hard to find routers without globally routable IP on a loopback or interfaces (whatever is used as the source for ICMP unreach in a given configuration) but if they have any IP at all, even a private IP, PMTUD still may work as long as ICMP messages with private source IP are not blocked by a firewall. But when a packet is dropped inside a carrier network (and not on its edges) because of a MTA mismatch it looks like a gross misconfiguration to me.

Isn’t it baked into IPv6 as something that happens automatically? I’m seeing better traction with ISPs seeming to finally hand out IPv6 addresses.

IPv6 still has PMTUD but ICMP blocking in v6 is less common because blocking of all ICMPv6 messages breaks network in more obvious ways than blocking of all ICMP v4 messages.

Absolutely. JF adoption is very poor and packet dropping is a (frustrating) thing. To be compatible, DCs need to invest in really powerful border/backbone routers (and their configuration) for fast enough packet splitting.

PS: Even at CERN, where cooperation is supposedly tighter, JFs were only an experiment, and, AFAIR, not a very successful one.

Reply on news.ycombinator.com