We use rsync.
When I worked for the local university we had to transfer data between machines to run experimental parallel programs on so-called "big data."
We used rsync.
YMMV, but if you want to improve throughput, consider carefully where your data has to go through. But rsync is rock-solid, well-understood, mature, and just does exactly what it is intended to do.
Scientific institutions that need to transfer large data sets have fast connections. :) How does 340 Gbps sound? Check out ESnet. http://newscenter.lbl.gov/2014/10/20/does-high-speed-network...
I heard about ESnet while consulting at Lawrence Berkeley National Laboratory.
Another option is to use a cloud data storage provider (Dropbox, Box, Google Drive, etc) and install their software that keeps files in sync. Then you can just put the file in a local folder and let their software sync it to the cloud. After sync is complete, you email a link to share the file. Of course, their software is probably just a GUI for rsync.
As for commercial providers - I've seen rate limits (never did it saturate the link) and size limits (0.5 TB will cost you extra). Moreover, the data will go through an intermediate hop, which is slow (Dropbox starts downloading only when upload completes, doubling your transfer time), plus your data is now...somewhere (which could be a regulatory issue, depending on the data).
- rsync
- SFTP-over-SSH, or SFTP over a dedicated VPN
- private, encrypted torrents
(Also, there's a patch for rsync that allows you to force computing the checksum in batch, not on each invocation; but that's getting into hairy territory that's rarely needed - I used it in exactly one case so far)
* Are we talking a few giant files or thousand of small-ish files?
* Who is on the receiving end? Technical people from who we can expect that they are able to run some command-line stuff or your average joe?
* What type of scenarios must the solution work? What OSes? Is it acceptable to install extra software or must it work out of the box?
etc. etc. etc.
This Google search:
https://www.google.co.uk/search?q=rsync+auto-restart+failed+...
returns this link:
http://superuser.com/questions/302842/resume-rsync-over-ssh-...
which contains this script:
#!/bin/bash
while [ 1 ]
do
rsync -avz --partial source dest
if [ "$?" = "0" ] ; then
echo "rsync completed normally"
exit
else
echo "Rsync failure. Backing off and retrying..."
sleep 180
fi
done
When the connection dies, rsync will quit
with a non-zero exit code. This script simply
keeps re-running rsync, letting it continue
until the synchronisation completes normally.
Easier, faster, and more flexible just to write the script. It's what I do.
For automated transfers ("retry until done"), I use the lsyncd wrapper (which also watches the source files, so you don't need to poll for changes: it wakes up by inotify).
--partial keep partially transferred filesIn other words, this feature is a prerequisite for auto-resume, not auto-resume itself (which can be scripted in ~10 lines, as shown in a sibling thread).
--append append data onto shorter files
--partial keep partially transferred files
--append append data onto shorter filesI asked why don't you just FedEx it? It's too important, the client said, and we know and trust you.
It was funny, I had it in a laptop bag, and didn't let go of it except to go through the security scanner... the only thing missing was the handcuff connecting it to my wrist. :)
Indeed, getting a trustworthy courier service is so hard that actually sending an in-house employee is worthwhile, even though their hourly rates make this extremely expensive: you are removing tens of abstraction layers, while preserving high degree of control ("Oh, we might have run it over with a truck. And accidentally put it on a plane to New Zealand on hop #3. And they can't seem to find it there.")