Is this part true? Aren’t the larger data center interconnects using 9000 bytes internally? Also, wouldn’t fat large distance links be using larger packets? I would have thought that a 4% wire overhead (and who knows how much CPU) would have pushed people away from 1500 for anything but last mile.
Path MTU detection is all sorts of fragile, so exposing a tcp maximum segment size implying a MTU above 1500 is asking for trouble; actually even just 1500 isn't always a good idea [1]. There are many networks where too big packets are dropped without notifications, and if the other end is also on a 9000 MTU LAN, then you can get stuck. sigh
On the host side, packetization is not that big of a deal; nics can help, but it's also just not that many packets. Where hosts tend to run out of cpu is when you're dealing with much smaller packets.
Larger packets could help routers of course. And larger packets would probably mean fewer acks, which would be nice too. But, it's unlikely to happen unless mtu probing is enabled in more places.
[1] The best is to send the lesser of (theirs - 28) or your actual value. Well best is relative, this will result in the most successful connections, at the expense of more packets for networks where people aren't insane.
1) Dropping all ICMP packets on the floor for "security" reasons, which means that basic diagnostics is now impossible, you get random issues with software that uses PING, and Path MTU Discovery is broken forever.
2) Leaving internal firewall ports with the default settings, which are intended for Internet-facing ports. For example, for "denied" incoming traffic from the Internet, the correct response is to silently drop the packet. Internally, the correct thing to do is to respond with a NACK to instantly close the connection. Without this, you spend days and days chasing down random and difficult-to-troubleshoot timeouts and weird 30-second delays all over the place.
As a random example, Windows RDP makes a HTTP call out to the Internet from the server to verify a CRL. This is safe and secure. Without this, bad certificates can't be blocked. Unfortunately, this happens in the "SYSTEM" context, which tends not get proxy settings applied to it, so it is often blocked, with a 30-second timeout. This failure is cached for 24 hours on the server, which causes a maddening delay when you connect to servers. Every day. Every server. But you can't reproduce it, because the second time it won't happen. (It also won't happen if anyone else connects to the server right before you.)
Years later, I still get angry remembering the snarky comments by the firewall guy saying that I'm just imagining things.
At high volume mtu bottlenecks, the routers dropping packets are likely to limit how many ICMPs they send; otherwise they'll run out of CPU. This wouldn't be terrible if ICMPs weren't so commonly dropped.
Some bottlenecked routers don't have globally routable IP addresses, and may not be able to send ICMP at all. This can happen inside long haul networks, where there may be tunneling that reduces the effective MTU. This seems less common today, but I dunno? One 'recent' advance is that some PPPoE networks use 'mini jumbo' frames, with an ethernet MTU of 1508, so that the PPP mtu is 1500; that may be happening on other layers with tunnels too.
PS: Even at CERN, where cooperation is supposedly tighter, JFs were only an experiment, and, AFAIR, not a very successful one.
It may sound simple but it was pretty hard for them to reproduce.
You are correct that intermediate boxes all take a perf hit for the extra packets, because routers and other boxes are usually packets per second bound at some point, rather than bandwidth.
This normally only happened when servers arrived with new NICs and no one thought to give us a heads up.
But when it comes to external L3 peering between 3rd-parties (eg: the Internet), it's very rare that people will mess with the IP MTU. Unless you can guarantee that the IP PMTU is higher than 1500 bytes (across multiple segments that you may not control, including ISP last-mile) there is simply no benefit.
The reason the Internet still runs exclusively on 1500 byte IP MTU is that any lower MTU in the path will effectively make the jumbo segments useless. This means any PMTU-D problems customers experience would have been unnecessary and avoidable by just using 1500 everywhere.
Many public IX's (peering exchanges) have a jumbo vlan but it's almost always separate from the standard one that only allows 1500 byte MTU.