Issue 87 – google-compute-engine – UDP Packet Fragments cannot be reassembled
code.google.com
code.google.com
If you want to play in that world you need to be prepared to handle MTU discovery on your own, or else design your app around deliberately small packet sizes.
That's not to say that this isn't a bug. But let's not start editorializing our titles: the apps for which GCE is "unusable" are buggy apps to start with.
We didn't "pick" UDP. We operate a VoIP related service that interoperates with many different carriers via SIP. Almost all of those carriers ONLY use UDP. SIP UDP packets can often be fairly large. This is especially problematic because GCE uses a non standard MTU size (1460 bytes). This does not make our app, or every other SIP related app that is forced to use UDP, "Buggy".
Rough summary of what I had here: I'm an engineer on GCE (in particular I built our current virtio-net device and a small fraction of the other fiddly bits that sit behind that) -- some details in the bug jumped out at me and I thought there might be a quick fix, but I hadn't processed all of the details and posted a bit prematurely. After further review my original post was essentially content free other than 'IP fragmentation works correctly between internal IPs', which is not germane to the actual customer-reported issue.
When I jumped on HN on my ride home I saw this and was a weeee bit over-eager to be helpful.
(Off topic: has anyone tried grafting a TLS handshake onto ESP?)
Interestingly Linux will actually, by default, throw EMSGSIZE any time you try to send() a UDP datagram that is larger than the detected network MTU to the destination. As I understand it, you have to explicitly turn this behavior off to get fragmentation.
Isn't fragment reassembly essentially implying all the packets (fragments) will arrive, and will be rearranged in order? i.e. exactly what TCP does?
For example imagine sending a single 1MB packet via UDP, letting it fragment and be reassembled. What distinguishes that from TCP?
No. It is implying that it does what UDP does. TCP is different because it constructs a continuous stream. UDP is inherently packet/message based.
UDP ensures that all of the IP fragments for a given UDP packet get reassembled and rearranged and order. The UDP packets themselves are not arranged in order, and it is possible to see duplicates.
> For example imagine sending a single 1MB packet via UDP
Not possible. You can't send more than a 65,507 byte packet via UDP[1].
Sigh... unfortunately, there are a lot of misconceptions about the basic plumbing of TCP/IP.
[1] UPDATE: as was pointed out in comments, my statement was only true for IPv4. Since IP packets in IPv6 can have jumbograms larger than 65,535 packets, IPv6 UDP does support larger packets. Of course, in that case, UDP doesn't handle the fragmentation/defragmentation protocol itself, and of course GCE won't handle anything IPv6.
https://tools.ietf.org/html/rfc2147
(Good luck with that over the public internet, but it is possible to make udp packets larger than 64K)
IPv6 jumbograms are a whole different kettle of fish, as they are really handled at the IP layer, and are supposed to only be for cases where you have an MTU > 65,575 bytes. As that link describes, the UDP protocol itself doesn't do any fragmentation or reassembly of those packets. RFC 2147 is really just an update to handle the fact that IPv6 can have a single packet that is larger than an IPv4 packet.
"Note: Compute Engine networks only support point-to-point IPv4 traffic. Broadcast and Multicast are not supported."
https://cloud.google.com/compute/docs/networking?hl=en
And this:
https://code.google.com/p/google-compute-engine/issues/detai...
So: Sorry!
It's a thought experiment not a specific number. I'm trying to demarcate some of the differences between TCP and UDP, and UDP reassembling fragments looks a whole lot like TCP.
No handshaking, no blocking, no resending, no throttling.
It's the same as a small UDP packet in every way except for a higher chance of packet loss.
There is only a tiny bit of similarity in that they are both dealing with reassembling a higher layer protocol payload from fragments split up over IP.
Let's count the ways they are different:
1. UDP does provide ordering guarantees within a UDP packet. It provides no ordering guarantees between UDP packets. Consequently, the protocol never has to worry about ordering more than 2^16 bytes of IP data (that makes the sorting algo really simple and efficient).
2. UDP doesn't have to deal with complex stalls or partial delivery of data. Either it has all the IP packets for a given UDP packet (in which case, it immediately delivers the UDP packet), or it doesn't (in which case it doesn't deliver it). If there is a UDP packet is missing a fragment, but a subsequent UDP packet is fully assembled, that one gets delivered.
3. There is no retransmission. UDP doesn't retransmit. It doesn't even guarantee deduplication. It is entirely possible that fragmented packets going different routes can end up creating an echo of multiple copies of a UDP packet being transmitted to the recipient.
The above makes UDP far simpler, and the contract application layers operate under very, very different. If you look at your typical TCP stack's logic for reassembling fragments, you'll find it far, far more complex than for UDP.
Are you sure? If some fragments show up and some not, doesn't it have to wait and see if the rest show up? Maybe it doesn't have to wait for a long time (like TCP), but it does have to wait.
Both point 1 and point 2 are a matter of quantity vs TCP, rather than quality. i.e. UDP+fragments is a sort of TCP-lite.
$ cat /proc/sys/net/ipv4/ipfrag_time
30
Much of the stall logic is tied in to whether to wait longer, whether to deliver partial data, whether to request a retransmit, whether you've already transmitted the data (once you reassemble a UDP packet, you can fire it and then promptly forget you ever saw it.. if defragmentation causes you to reconstruct and send the packet again, that's 100% okay!) and most importantly, what to do with an open ended amount of data that may need to be held up while waiting for that one lost fragment.
The logic is simple:
1) Allocate space for UDP packet. 2) Fill in UDP fragment data as it arrives. 3) If all the data for a given packet is there, transmit it and delete all memory of the fragment. 4) If you timeout, delete all memory of the fragment.
That is so, so, so much easier.
If you looked at the code, you'd not think it much similar.
UDP has no feedback from the receiver to the source. If a packet is dropped, the IP stack on the receiver will never complain and the IP stack on the sender will never resend. SO the receiver will silently drop the whole packet.
Notice it can be hard to tell if a packet is dropped, or is reordered. You receive 2,3 - maybe 1 is still coming but reordered? So detecting dropped packets is a heuristic, depending on timing. Various algorithms will use a timer to say when to give up on a packet, or drop a packet if a frame from a subsequent packet is received, or tolerate one frame from a subsequent packet (maybe the last frame of the previous packet was reordered with the first of the next) and so on. But all the work is on the receiver. The sender is not involved.
Unless there is some application-layer protocol that notices packets arriving with a gap, and sends something back (that might also not arrive) like "I got packet a,b,d but missed c". Etc. Notice the application cannot resend frames; only packets because it is unaware of fragmentation/reassembly which happens in the IP stack.
Large UDP packets have a packets success rate of the single packet success rate raised to the number of fragments power, so don't get attached to them in situations with significant packet loss. But this behavior is just an unaddressed bug.
That's not entirely fair. UDP is actually a very convenient transport for packetized data of 65,000 bytes or less.
> You picked UDP because you have latency requirements that cannot be met by TCP.
That is not really a good reason to use UDP, as you won't necessarily get better latency (as has oft been demonstrated).
You should pick UDP because you have a connectionless protocol and/or one with data loss requirements that are different from those built in to TCP.
> that means you need to know what your packets are actually doing, and that includes fragmentation.
UDP is supposed to abstract IP fragmentation issues for you.
> If you want to play in that world you need to be prepared to handle MTU discovery on your own, or else design your app around deliberately small packet sizes.
I've used UDP based protocols all the time. I'd agree that I've had to be a bit more aware of path MTU when it comes to diagnosing or debugging problems, but in terms of the application code reading and writing UDP packets, I've never done MTU discovery.
> But let's not start editorializing our titles: the apps for which GCE is "unusable" are buggy apps to start with.
Considering there are bytes in the UDP header specifically designed to allow you send packets larger than MTU size, this is actually a huge problem. For many people, this requires reimplementing UDP's fragmentation & defragmentation logic up in layer-7.
Isn't it bad enough that people are reimplementing TCP on top of UDP? Do we really need to tell them to reimplement UDP on top of UDP?
Nobody asked for that, just to send slightly larger packets that will probably get through.
> If you wanted simple transport of large data chunks, you would have chosen likewise.
Why? A 4KB data chunk doesn't need TCP more than a 1KB data chunk.
You know the 1500 byte MTU is completely arbitrary and many networks set it higher, right?
> You picked UDP because you have latency requirements that cannot be met by TCP, and that means you need to know what your packets are actually doing, and that includes fragmentation.
Fragmentation itself is a cost of microseconds. The main problem with TCP is head of line blocking, and moderately large UDP packets don't have to deal with that.
You don't need to trace every packet all the way through the network to get low latency and manually handle loss.
> the apps for which GCE is "unusable" are buggy apps to start with
Nothing you listed is a bug.
A key principle of UDP is quite the contrary of what you are saying. UDP applications can have guarantees that they don't have to engage in MTU discovery or fragmentation issues. It provides a way to have an abstract, static contract about packets that is agnostic to layer-2.
Because UDP is comparatively simple, it has been abused as a proxy to implement your own protocol on top of IP, and in that context you of course you really have to deal with al those concerns. However, it is a terrible mistake to think that is what UDP is about or how one should use UDP.
If I was Linus, now would be my cue to unleash a tirade about how you don't fing break userspace. Ever. Protocol layering works because higher level protocols depend on long standing contracts with the lower layers. Google decided to willy nilly go and break the basic contract of UDP across their entire cloud and the top comment here faults protocols built on those guarantees. Disappointing.
> Apparently the update from Google is to find another support channel to escalate, or only use TCP.
I've never had a Google product issue ever resolved using an official channel I was directed to. It's only by back channels, friends of friends, posting to HN, etc.
I do not have experience with GCE, but saying that you don't get support for any Google product is disingenuous.
https://forums.aws.amazon.com/thread.jspa?messageID=536049
https://www.reddit.com/r/aws/comments/3ccn5o/real_ipv6_suppo...
SCTP is ideal for this use case but it is not well supported by OSs or networking APIs. TCP works but adds overhead. TFTP works, is UDP-only and has less overhead than TCP, but it does not respond well to packet loss. UDT is like TFTP done right, and is a good solution if you can setup a dependency on its large C++ library.
Last I checked UDT was far more complex than UDP, and since it is layered on top of UDP, I'd think it'd be vulnerable to this problem (although it had all kinds of logic for correctly sizing packets and windows, so maybe it correctly avoids this problem). Either way though, from an application perspective UDT looks much more like TCP than UDP, so I wouldn't think it'd be an obvious choice to replace UDP.
You must never make assumptions about what fits in one packet. The MTU could be 100, or less, or 8000, or more.
As soon as you start doing math based on MTU values that you don't permanently have end-to-end control of yourself, you're setting yourself up for trouble.
The submitted title was "Warning: Google Compute Engine Unusable for Applications Which Rely on UDP", which several commenters have objected to as exaggerated.
The info I have came from a conversation with an AWS Solutions Architect.
It's a couple of years old, but based on my experience it's still true today.
We were using a Non-AWS DNS resolver (aka Google) and we would often get dns resolution errors despite our NAT not being remotely taxed by the traffic.
(Of course this assumes your protocol has some alternate method of verifying transferred data, which many do.)