Pushing the Limits of Amazon S3 Upload Performance | Hacker News Reader

Pushing the Limits of Amazon S3 Upload Performance

improve.dk

89 pointssorenbs17 comments

I have seen some people get performance gains from using UDP. TCP tends to back off rather aggressively after even a single dropped packet, and the connection is slow to get back to speed. With UDP, the uploader can just stream packets with sequential ids, and the receiver can respond whith a negative ack if it misses a packet in the series.

UDP Data Transfer (Open Source): http://udt.sourceforge.net

Bandwidth Challenge: http://www.hpcwire.com/hpcwire/2009-12-08/open_cloud_testbed...

Aspera: http://www.asperasoft.com/

TCP backs off aggressively to avoid congestive collapse of the network. It is backing off in order to preserve shared infrastructure. If everybody used your UDP trick then the Internet would literally collapse. This is detailed at http://en.wikipedia.org/wiki/Network_congestion#Congestive_c... . You should respect the rules of the road.

You can (and should) set a bandwidth limit with UDT.

What do you set the limit to? What do you do when a router fails and your available bandwidth suddenly drops? All of the TCP end hosts will backoff properly and avoid melting the network with endless re-transmits. All of the TCP flows will converge towards sharing the available bandwidth, even if that is a moving target. Applying a congestion control algorithm on top of UDP like the other response says is fine. Perhaps TCP's specific congestion control scheme isn't what you need (it sucks at live video for example). But you really do need a congestion control algorithm.

I'd like to add to this by saying that the new congestion control algorithms ought to be "friendly" to TCP to avoid congestion collapse. Backing off by a factor smaller than 2, or increasing additively by a delta more than 1 MSS per RTT are all tricks that would cause harm to normal TCP flows.

Granted that TCP isn't really "fair" between flows that have different RTTs, one could really justify tuning their TCP behaviour. But the challenge is to do this _automatically_, which is what TCP has so successfully done for the past few decades.

EDIT: DCCP decouples congestion control from reliable delivery and its congestion control algorithms are TCP friendly.

I'm pretty sure UDT does congestion control.

Isn't this ultimately up to the congestion control algorithm that is used alongside of UDP? Aspera and UDT, for example, have tunable congestion control that can be more or less greedy.

I'm not a protocol hacker, so hopefully one will weigh in here. But I do remember some Sky Is Falling discussion over uTorrent's use of UDP for transfer, and the uTorrent line was always that they were implementing UDP transfer in a way that played nice with TCP.

uTorrent's UDP congestion control algorithm (LEDBAT) goes further than playing nice with TCP. Unlike TCP, which only responds to packet loss, LEDBAT also responds to delay. This makes it yield remarkably quickly to anything else that might use the link.

Also: http://tsunami-udp.sourceforge.net

S3 does not support UDP uploads.

Assuming that EC2 is on a shared network with S3, you could do something like the following:

CLIENT <-UDP-> EC2 Instance <-TCP-> S3

Not certain if it is worth the hassle, though.

http://www.asperasoft.com/en/misc_content/default_1/Aspera_S...

Pretty expensive though.

As seen in the tracerts that I posted, while S3 is not on the direct subnet of EC2, it is certainly much much closer. However, without having tested it, I am rather certain that the added complexity and latency of putting an EC2 based UDP proxy in betweeen the colo servers and S3, probably wouldn't be worth it.

Timely! I'm just in the process of uploading 32 million objects to S3 :-)

The parallel upload code I'm using is written in Python, using the multiprocessing and boto libraries and is here:

http://github.com/twpayne/s3-parallel-put

It has some nice features, like reading values directly from uncompressed tar files - this means your disk heads will scan linearly rather than seeking around. It can also gzip and set Content-Encoding, restart interrupted transfers from its own log files, and do a MD5 sum check to avoid putting keys that are already set.

Comments and feedback welcome.

Sadly no mention of multipart upload. If you use S3's multipart upload feature you can get similar performance for one file by uploading multiple parts of the file at once. I saw a substantial throughput improvement when I implemented some multipart upload code, even when uploading from EC2.

Totally true. However, I had to limit the scope of this post somewhere, and in this case, I focused on multithreaded upload of smaller objects, as mentioned in the first paragraph. We do have some larger object uploads, though they are a minority. I may do a separate followup post on high performance single file uploads through multipart uploading.

Wow that's amazing. I didn't realize you can utilize beyond 10Mbs in EC2.

OTOH do people ever delete objects from S3? It seems people keep uploading to S3.

Reply on news.ycombinator.com