Ask HN: I need to transfer 10TB of Data across Pacific | Hacker News Reader

Ask HN: I need to transfer 10TB of Data across Pacific

6 pointsstkim118 comments

I just have no idea what to use. Would there be anything better than carrying ten 1TB HDD myself?

The best solution depends entirely on your optimization conditions.

If transaction time is important, hard drives are high bandwidth. Ten 1TB hard drives are $500. Two 6TB hard drives are also $500, but you have fewer drives to worry about shipping. Five 2TB hard drives are about $370, if drive price is more important. Using ten 1TB drives is only best if we're optimizing for load / unload time and have that degree of parallelism in the pipeline. Otherwise, one of the other two will be better, depending on price sensitivity vs. other factors.

If it's not as time-sensitive, something like BitTorrent Sync will get the job done. This doesn't require shipments, additional hardware, or mucking about with hardware at either end, so it's very nice if "get it there ASAP" isn't a constraint.

Regardless, don't ship the only copy.

Thanks for the concise summary.

Mail it to me at the hacker house in Kansas City. We do uploads for people all the time using Google Fiber. Probably take a few hours (YMMV). Google Homes for Hackers. My cell is right there on the site.

+1 for being awesome - but the OP mentioned 'across the Pacific' - if he's got to transfer 10TB to, say, Sydney or Beijing, I'd bet your upstream won't be the limiting factor, it'll be the congested peering ports and high latency links between the States and APAC providers.

That said, nothing ventured, nothing gained. Maybe try a speed test between the hacker house and the intended AS before anyone ships a drive anywhere, and see if the other end can keep up with you?

I really appreciate your warm gesture. I wish my neighbors are just as kind as you are, sir.

http://filecatalyst.com/ http://filecatalyst.com/open-source-fast-file-transfers/ http://www.jscape.com/blog/bid/78782/How-to-Boost-File-Trans... http://asperasoft.com/ http://www.signiant.com/

https://what-if.xkcd.com/31/

See? this is funny. Some says this is the age of data. 10T is just a tiny amount of bits for some entities, and I cannot reliably transmit that much across pacific other than shipping actual HDD. :(

Thank you for the comic.

Two 6 TB or three 4 TB is probably a better choice than ten 1 TB. Although you may want to have an extra disk so one could totally fail in transit; so maybe six 2 TB. Build a big parity archive (par2 or similar) first. If you have 1gbps end to end, it would take most of a day (make sure you have big TCP windows)

With long ping times you want to split your upload into many parallel parts so the odd dropped packet doesn't bring things to a halt. Then it just depends on your bandwidth.

We were able to saturate the gigabit nic on an ec2 instance in Ireland uploading to S3 in Oregon. At that rate a 10TB will take a little over a day.

I'm puzzled. Can you explain what you mean by bringing to a halt?

Do you mean a dropped packet might terminate the TCP connection; or at worst result in the timeout of a segment and a return to slow start? Nevertheless, I would have thought a timeout is a rare event at best, even on high latency links, due to the post-Jacobsen RTT estimator and fast retransmit kicking in at 3 duplicate ACKs?

5 x 2TB HD's bought at Costco + shipping charges.

Total cost: <$1k

Rather small package. "Upload" time <10 days.

Whether you can use distributed web services like s3, elastic mapreduce, or google storage to copy it largely depends on what kind of ISP connection you have on both ends. With gigabit and multipart transfers it seems feasible within a week.

Sneakernet (https://en.wikipedia.org/wiki/Sneakernet) is likely the best choice.

Bit torrent it to the other side of the pacific. With a 35mbps connection will take like a month.

Mail a HD.

FedEx or similar

Globus.

Reply on news.ycombinator.com