Hacker News Reader

Comment by lobster_johnson | Hacker News Reader

I always wondered why nobody used UDP for bulk data transfer: Ship all the chunks over with a sequential number identifying each, then when you've reached the end, have the client request any lost packets, then repeat the cycle until all are transferred. After all, the client doesn't need everything in sequential order if it knows the size, and it can deal with holes as long as they're eventually repaired.

This gives you (in theory, I've never tried it) better throughout, and you get the equivalent of TCP's retransmission without the window semantics, since you move all the retransmissions to the end. On the other hand, it's probably bad for everyone else, since there's no congestion control.

View on HN

> ship all the chunks over with a sequential number

Except that if you ship them too fast, they will at best simply build up in a buffer somewhere (e.g. near a bottleneck link), or more likely they will overflow causing many of your packets to be lost. So not only did the recipient not get them (and you have to figure this out and resend them later), but you also wasted a bunch of resources sending packets only to have them dropped.

And if you send them too slow, you are leaving some potential capacity on the table. So you need to send them fast, but not too fast.

So you add some additional logic to implement flow control and congestion control. And that's TCP, pretty much.

Exactly. The core feature provided by TCP is not reliability, that is trivially easy. What TCP gives us is robust flow control that can detect and adapt to current network capacity.

A robust flow control that's tuned for bulk downloads, and badly adapted for small real-time streams.

Sure. And not even the best known flow control mechanism. But nonetheless that is the important innovation of TCP, not reliability. The post by lobster_johnson upthread implied that you could use UDP for "bulk data transfer", which basically doesn't work without reinventing something akin to TCP.

CNN has a video transfer tool for b2b subscribers that uses multiple UDP streams to increase transfer speeds. Instead it saturated and brought down our company firewalls, which were older and our main bottleneck. All for just a potentially few more megs of speed on TB sized videos. Gotta be careful with UDP.

That just sounds like your firewalls are made of pentium pros and duct tape. TCP can oversaturate a connection too, and a firewall should be able to handle large packets at line rate.

Yeah the firewalls were ancient and already scheduled for an upgrade. This helped accelerate the schedule for sure. The company had about 800 employees doing all kinds of things, watching Netflix, watching videos on YouTube and Facebook, downloading whole ISO's, not to mention many business processes transferring files around, but the second one person started using this video downloader tool it brought everything to a crawl.

> a video transfer tool for b2b subscribers that uses multiple UDP streams

That sounds like FASP (http://asperasoft.com/technology/transport/fasp/)

I believe something like that was the underlying tech. I researched it at the time but I just can't remember 100% now, although I think that was it. It actually was implemented via a Java applet in the browser, so there was no way we could block or restrict a specific exe with group policy. The firewalls were already scheduled for replacement/upgrade, so for a few weeks we ended up just watching traffic and slapping wrists if more than one person ever tried to use the tool at the same time.

There is actually such a protocol - UFTP. It was designed for high-latency connections such as satellite links. It can use multicast to allow multiple recipients over such a link. If memory serves, it was designed by Gannet to help transfer news content nationally for USA Today.

I've tested it on "long fat pipes" when evaluating it for a project a number of years ago. The numbers don't sound that impressive now. I was able to saturate the 100Mbps NIC of the slower server in the transfer. The transfer was from Northern VA to LA over the Internet. This was from data center to data center and the network capacity was such that I knew that the transfer rate would not cause any serious packet loss.

Using simple TCP over the same link, same computers was limited to around 40Mbps. More modern TCP implementations have a larger window and are able to better saturate high-throughput, high-latency links.

UDP messages are limited to 64K. It can be a pain to reassemble. lots of ugly corner cases.

We had need to send one type of message split due to size with udp. We were on the same subnet sending and receiving so it worked almost always. We were also using multicast.

Its very very hard to figure out whats going on when things don't work ( You don't know if the control messages are getting through, when you ask again or if the message just isn't being sent). At some point you are reinventing tcp/ip.

I always liked the way udp messages are received though, all or nothing. Not the byte stream that is tcp which needs to be broken into messages manually.

Picking a smallish, fixed chunk size should work, and it could even be made to self-regulate. Reassembly is trivial -- allocate the file first, write chunks to their right location. And for the control stuff (i.e. requesting chunks) you just use a TCP connection. UDP for the data only.

If you do the obvious work to convert UDP to a reliable streaming mechanism, you will generally more-or-less reinvent TCP, only probably more poorly, and would probably be better off just using TCP. UDP is a good choice when you have options or requirements that TCP can't accommodate. For instance, the classic case of streaming audio has both; it both requires timely delivery and has some very sophisticated techniques for dealing with missing packets with a lot of local intelligence about human perception and such built into the codecs. If you can drop packets without issue, if future packets completely supercede past packets (FPS network gaming for instance), there's a lot of possibilities, but if you're just going to recreate reliable streaming you probably be net losing on account of the effort spent doing that instead of something useful, and bugs in the implementation.

Why are so many comments here ignoring the purpose of the article?

The purpose of this is for games, not web traffic or torrent2.0.

In real time games where you're syncing physics states or player locations data >200ms isn't just useless it's actually negative.

This article is 2+ years old, and is the foundation I suspect for libyojimbo which hybridizes UDP low latency communication for large player real time games and reliable messaging (basically custom TCP?) for less time critical things.

I don't know, but that doesn't have anything to do with my comment, which is entirely technical. Perhaps you meant to put it somewhere else?

You've kinda answered your own question there, congestion control becomes a big issue(also MTU discovery to a lesser degree).

Games sit in this happy space where the data rate is ~10-20kb at most so they don't need to worry about congestion control as much.

Additional information: The source engine (valve) has been limited to 30 kb/s for more than a decade.

There are many protocols that use this approach, Aspera FASP is possibly the most advanced one. It uses a TCP control channel to manage the rate on the UDP data channel.

On LAN, this is basically how Symantec Ghost works; it multicasts the image over UDP, and each client figures out what they missed. Those chunks get retransmitted at the end.

> I always wondered why nobody used UDP for bulk data transfer

That would be odd. It's not true. There's TFTP all the way up to UDT.

Although to be fair, given that with TFTP each packet must be acknowledged before the next one is sent, you're actually achieving a way worse throughput than TCP. Not a very good example on how to use UDP efficiently IMO.

UDT is a better example.

http://udt.sourceforge.net/

What it really comes down to is file block size granularity. TCP is simply using a much finer block size (~1500 bytes) which means a higher likelihood of arriving than for a large UDP packet. Furthermore, you'd still have to implement your own checksumming algorithm to ensure data integrity as UDP doesn't require a packet checksum for IPv4. I suppose you could offload file-block integrity validation to another thread so you can receive and verify in parallel, but at some point you have to checksum every block anyways which means the full file must be processed to figure out where corruption might be before sending retry requests for the missing packets.

I see what you're getting at by trying to avoid processing the packets in order if it wastes time, but ultimately I think its a micro-op that introduces other bottlenecks. That's why bit-torrent still typically uses TCP even though they could potentially receive file parts in any order.

But bittorrent breaks the files in to chunks it can receive out of order and sends those over TCP, so TCP only preserves ordering within a chunk. It might even get chunks from several different people.

The real answer is that chunking over TCP is efficient enough that little is gained by chunking over UDP.

Yes what I'm saying is that bit torrent uses TCP to segment fileblocks instead of using UDP to send the file block as one large datagram

That sounds like a TCP congestion control mechanism, except worse! TCP leaves much to be desired in terms of performance, largely because their congestion control fails miserably on lossy links (TCP RENO). There are better backoff mechanisms that can be added on many distros, like Westwood. And then there are altogether more efficient protocols like QUIC. But designing one for general purposes from UDP is easier said than done.

NFS on Linux defaulted to UDP for what seems like a really long time. There were perf issues with TCP (can't remember what they were).

NFS used to default to UDP for the same reasons that games still would use UDP, it just needed the data as quickly as possible since it was blocking IO. Now a days, we don't need UDP for NFS because networks are a lot faster and the advantages of TCP start to over rule UDP.

The original reason was that using UDP was much faster CPU-wise back in the 80's. There was a requirement to go fast because it was used for transferring data in bulk, not events like games.

When Linux came around it just followed suit. TCP support came to NFS only later with other bells and whistles in later NFS versions, and Linux stuck with basic NFSv2 for a good while.

NFS was designed for local-area networks, so it didn't need to worry about working over the Internet.

In addition to the network being much faster relative to CPUs back then, the TCP stacks hadn't seen the tuning they have now. Making TCP fast came later.

NFS (was) stateless and idempotent. Lose a packet? Send it again! Which is mostly what you want, since there can be dozens of processes performing IO and they're all independent. Last thing you want is head of line blocking.

> I always wondered why nobody used UDP for bulk data transfer

1) Bulk data transfer doesn't work over UDP.

There is no flow control whatsoever with UDP and no guarantee, it sends stuff without distinction between a 56k uplink and a 10Gigabit LAN. The packet loss and lack of reliability are atrocious when you send non negligible amount of data.

2) You'd need to create a protocol on top of UDP to handle the basics (controlflow/error/retransmission), that's equivalent to re-inventing TCP. Don't re-invent your own TCP, just use TCP ;)

>1) Bulk data transfer doesn't work over UDP.

Tell that to all of the people that use TsunamiUDP to do bulk data transfer. It uses TCP as a control protocol, but the data transfer is all UDP.

It sounds like they would agree.

Bulk data transfer doesn't work over UDP, but it can work over UDP and TCP.

The data transfer itself is happening over UDP.

UDP is usable for this when used to transport a file transfer protocol. I've seen for instance Kermit over UDP. Basically, if you transport data using a protocol that already takes care of integrity, using TCP is more or less redundant and just adds annoyances (connection management).

Back in the day of hard drives, the random access I/O would be nasty and you'd have to be careful about creating sparse files, so you'd end up double writing the missing packets. you'd also still need congestion control so might as well just use TCP to retransmit.

Bad for everyone else, and it's up to the implementation to handle flow-control. Congestion-control handles network related loss/latency/load. Flow-control helps with receiver related loss/latency/load.

A lot of the internet's bulk data transfer uses UDP, torrents.

BitTorrent is mostly TCP these days, though, as pointed out elsewhere in this thread.

In practice, I think you'd end up having to reimplement most of TCP on top of UDP anyways.

Reply on news.ycombinator.com