Hacker News Reader

Comment by zokier | Hacker News Reader

Is there any place for IP fragmentation anymore? On surface level look, the main motivation seems to have been fitting over-sized dnssec responses in single datagram, but with ecdsa/eddsa is that still a relevant concern? I realize some TXT etc data might be big, but presumably those could also be queried over TCP/HTTP/QUIC/etc; I see UDP more as the fast path for common A/AAAA queries, and those should fit in unfragmented packets just fine?

What are other cases where fragmentation truly makes sense?

View on HN

There has never really been a place for it; "Fragmentation Considered Harmful" is one of the original and most famous "Considered Harmfuls" and it's from the late 1980s. A lot of protocol engineering, from MSS options to ENDS0, goes into ensuring that you never hit the fragmentation case in the first place. It's been kept around as a mechanism of last resort for weirdo hops with bizarro MTUs.

I’m surprised by the willful disregard of the feature, e.g., with some routers just dropping fragmented packets. Sending packets bigger than the MTU seems like a reasonable thing to want to do—especially given that IP by design won’t guarantee a stable path between endpoints. Was the reasoning that it’s always better handled at a higher protocol layer?

Yes, that's the reasoning. Fragmentation is kind of a performative half-measure. You're not really enabling delivery across varying network links, in that the performance is so bad (necessarily!) that it alters the service model for many protocols.

IPv6 moves to an end-to-end fragmentation model, but even then the right answer is to negotiate in a higher level protocol a maximum segment size for the path you're talking on, and then just avoid fragmentation entirely. Fragmentation is an absolutely wretched stream transport protocol!

the messy challenge here is that the trade-offs shifted, but persist contrarian arguments. packets are too small to be efficient without doing excessively tight loop optimizations, which is what makes fragmentation "slow" (it's not such a big slow down in a simple software implementation, it's a disaster in a highly common case optimized device). On the other side in order to move away from those excessively optimized systems while still delivering on user demand for higher throughput we need larger payloads - but we can't get to larger payloads without transparent solutions for fixup (ICMP, fragmentation, etc). Sprinkle in some stuff that has never been formally fixed (bad parts of ICMP concerns), and it's an ongoing recipe for ossified non-progress.

Fragmentation isn't slow because routers are bad at fragmenting (though: they are), they're slow because the loss of any one fragment forces the discarding of the whole packet. Because you can't possibly make a transport protocol as dumb as fragmentation reassembly fast, forwarders don't optimize it.

The picture in the case of losses is pretty complicated with dack, sack, congestion control, and so on. If loss is high enough that this is a main concern TCP performance is generally shot already.

I'm not a network engineer, but somehow I imagine that for at least some links fragmentation could be also handled transparently at lower layer, i.e. limit fragmentation to single hops and always reassemble packets before passing them on.

It is. In my opinion that's what you should do if you can't provide 1500 MTU to your users directly.

How is fragmenting and reassembling a better option than just communicating what the real MTU is so higher-layer protocols can adjust to it? What is the extra mechanism buying you?

At this point lots of things don't work right otherwise, and assuming your users want to run wireguard over this link that takes them down to 1420, and if you're already at 1420 it takes them down to 1340... Anyway, you can probably use a lot less header overhead if you do it at a lower layer too.

Automotive ethernet, particularly for high data rate sensors like LiDAR or radar in autonomous vehicles have a predefined 1500 byte fragment size and then expect fragmentation of larger datagrams to be done in sensor firmware. This prevents clogging the routers with huge packets and preventing time-critical messages from waiting in a queue too long while a large message sends.

But why do it on IP level instead of higher level like everyone else is doing it?

Why do it at the high level if the protocol supports it? Especially in an automotive system where you don't need to worry about rogue configurations as much.

quite the opposite:

CAN Injection: keyless car theft

https://kentindell.github.io/2023/04/03/can-injection/

I don't follow.

I meant why not leverage fragmentation at the routing+transport protocol layer if configs can be controlled to support it and not drop fragmented packets.

What advantage is there doing it at the application layer?

I'm aware that all the layers can be messed with by rogue malicious actors.

Though IIRC CAN doesn't really have a transport or routing layer; the ARB IDs are baked into the physical protocol, which is very cool, but it's fundamentally a bus architecture, and I'm not sure how it would apply to current thread.

that concern with these numbers is very surprising, do you know if there’s somewhere with writing about this?

No, unfortunately I only learned about it while debugging a very annoying issue that resulted from mixing automotive modules with off the shelf data center type managed switches and had to consult one of the system design technical leads, so I only got what detail he provided, but the intuition matches some of what you can get from understanding CAN.

I’m just a little surprised as economies of scale have pushed support for larger frames into most of the hardware you can purchase. I can absolutely believe somebody saying this, but intuition is telling me that it’s a misdiagnosed problem - if it isn’t there’s something interesting I’d love to hear about!

That's for throughput not latency, in automotive/realtime systems, you care about the latter and only slightly about the former. The real piece to that is that the modules themselves have to do the fragmentation, so their send queue can reorder things since each message is "pre" broken up, meaning that something that has a lot of sensor data and also some keep alive signals and some safety critical signals can interleave the safety and keep alive with the fragmented raw data by prioritizing at the send queue level, whereas if the data was being fragmented at the protocol level, that would potentially clog the queue when something even more time critical should be going out.

in a lot of these systems I interact with they're limited by packet rate more than packet size. I'm not disputing latency sensitivity in the application, I'm surprised that frame size in the normal range has measured to introduce the kinds of problems you're describing. The cost of per-packet parsing and reassembly is typically a higher proportion than would lead to a need to strictly pin to 1500 byte ethernet frames.

now if you have one system trying to send really oversized frames constantly, creating lots of fragmentation, and you have limited sized buffers, I could see a situation where the fragments are too regularly eating up buffer space, but that's even a slightly different problem too. I would expect inexpensive hardware to be able to handle 9k packets without introducing queue delay.

I heard upcoming networks are heading over 10gbps, which means their packet processing capabilities are necessarily well over 1Mpps, more than sufficient for latency concerns in the kinds of control systems hanging off of this.

Looking up some basic (possibly inaccurate) specs for the kinds of lidar used, frame samples won't fit in a packet, so we're talking in the range of 500 packets at 1500 bytes, or less than 100 packets at 9k, at around 30hz per sensor?

Clearly the deployed system works, we're dealing with 1500 bytes in a lot of places where it being the standard introduces a lot of inefficiency/hard optimization work, so it's hardly abnormal. As I said originally, I'm curious about what the factors are, they seem surprising.

I suspect, but cannot confirm or even really look up, that they cheaped out on some of the switch infrastructure in the middle. Also, at least some of the radars I dealt with produced object data and that meant that they had a defined "frame" size of 65536, even though it would often be mostly zeros, so it had to do fragmentation no matter how you set up jumbo frames and that needed to be handled at the sensor firmware level. There's also some tradeoffs inherent in gigabit speed T1 automotive ethernet and some stuff that was defined at the mac layer rather than the packet layer. Happy to chat more about it, feel free to look up my email in my profile.

> with ecdsa/eddsa is that still a relevant concern

Perhaps not, but with quantum-safe signature algorithms on the horizon, signatures are about to get much longer again.

WireGuard!

I had multiple issues with WireGuard related to MTU/fragmentation. Once, service reported that a system is pingable, but the management website didn't load. I was able to SSH into it, but similarily, the connection broke everytime I tried to run `ip addr` or view logs. Commands with short output worked, however – apparently, somewhere on the path the packets got fragmented and/or dropped without notice.

This is going to be true for any VPN - even if it were purely IP in IP encapsulation, you'd need an additional IP header to get the packet to the VPN endpoint, so your payload is smaller. But fragmentation probably isn't the right answer (especially since any packets marked DF will be dropped). If the MTU for your Wireguard interface is set correctly then anything trying to push a 1500 byte packet via a Wireguard interface should get back an ICMP packet telling it that that won't fit and adjust its packet size downwards appropriately.

I actually hit this recently, where I wasn't able to stream videos from certain sites over Wireguard. I spend a while with tcpdump and figured out that they were using Fastly as a CDN, Fastly was sending 1500 byte packets marked with Don't Fragment, my Wireguard endpoint was returning a message saying that the maximum packet size for the link was 1460 bytes, and Fastly was then… sending another 1500 byte packet marked Don't Fragment. To their credit when I was able to provide their engineering with logs showing this was clearly a Fastly problem, they fixed it fairly promptly

Did they fix it just for you or for everyone? A proposed project at work involves using Fastly as CDN and I'm curious if this is something we need to keep in mind.

Everyone, as far as I know

In my case, the application was using the Wireguard MTU correctly, but the Wireguard packet itself was too big for something on the path. So, the Wireguard MTU wasn't set correctly, basically. But Wireguard packets aren't marked DF, so one would expect a worse, but still working connection in that case. However, that something on the path seemingly just dropped the packets without ICMP notice.

Very interesting, I bet this had come up before but remained undiagnosed, yay for packet captures

> To their credit when I was able to provide their engineering with logs showing this was clearly a Fastly problem, they fixed it fairly promptly

regarding credit; did you get paid for doing their homework?

If your path requires fragmentation to get to the wider world, there's a good chance it simply won't work. There's a small chance that it will work with some destinations. I don't think there's really a place for it in that sense.

Maybe for UDP for protocols where the designers were really optimistic. IMHO, for DNS if you want it to work as much as possible, your queries and responses should fit in 512 bytes, or some people are going to have problems on some networks. DNS over TCP doesn't work everywhere, even if it's supposed to.

It's more palatable to require DNS over TCP or working UDP over fragmented IP if it's for big txt records like for mail, but if your A and/or AAAA records don't fit in 512 bytes, expect lingering problems that are hard to track down.

well, if you read the article. the issue isn't really fragmentation as a mechanism, its finding the appropriate (dynamic) path MTU. that's kinda gonna be a problem forever given the stateless nature of IP. it would be fine if people hadn't decided that ICMP was .. undesirable.

I guess you could just say '1480 for all time' and be done with it

Reply on news.ycombinator.com