Not only it works much faster than scp or sftp, but it also does not lose any file metadata.
I have not verified if newer ssh versions have corrected this, but a few years ago scp and sftp failed to copy extended attributes and on some file systems they truncated timestamps.
Looking at my terminal history, scp for a single file, rsync (which you don't need to pass that --rsh flag to.. during the 21st century ;P) for anything that requires the ability to be interrupted and resume gracefully (which is most cases), and, in the rare case when I need speed above all, the "tar lzop ssh bash pipe" trick (I don't know if it has a canonical name) is correct.
Isn't this what we all do?!? If there's a better way, I don't see how, but would like to know.
In that rare situation, I typically use tab completion to find the remote file. However, I bet that would SUCK on a high latency or low bandwidth link, where sftp would be just fine.
Solid Explorer (Android app) connects via sftp to Linux machines just fine. While I use Termux and rsync to do backups and bulk transfers, Solid Explorer allows me to rapidly browse remote files.
But yeah Rsync is a good choice.
Back in 1999, in one of my first professional gigs, people were still using ftp for sensitive data for automated data transfers.
Rsync+ssh would be great, but enterprise IT is slow to change.
AWS has hosted sftp backed by S3, and it's going to stick around for a while.
We're moving customers from FTP to SFTP. Most recent one was a well-known Fortune 500 company, which uses it to exchange some ASCII flat files with our software.
Mostly it's for XML files or PDFs.
It's simple, it works and it's easy for support to poke and prod.
Come to think of it, I once even set it up at a job in 2009. We needed to digest huge amounts of flat files from clients with little tech savvy. I put FTP on a server, put that behind OpenVPN, then sent the customers a pre-configured OpenVPN client installer. Never broke.
Tangentially, all this mention of FTP being used by banks and financial institutions is inspiring some choice nmap scans to run... ;)
I use rsync+ssh a lot, and it doesn't exactly work perfectly on long-distance 10-100 gigabit networks: ssh has a buffering problem at high bandwidths which the maintainers won't accept patches to fix.
Using scp or sftp would limit the transfer speed to a value several times lower than rsync in any case.
On short distance 10 Gb/s Ethernet I did not see problems, but it does not surprise me that on long-distance links they appear.
In such cases, if full speed is desired, it is likely that ssh must be replaced with either IPsec or a tunnel over TLS.
I don't know anyone in the HPC world who uses "IPsec or a tunnel over TLS".
This is interesting, can this be the reason for why playing 720p video in Firefox run with ssh and X11 forwarding is fine bandwidth-wise (easily 1Gb/s) but it causes responsiveness to mouse/keyboard actions to go into abysmal 5-30s territory?
Any references to the ssh buffering problem out there?
The first is the license. Because rsync is under the GPL, it will never be bundled in the OpenSSH.com distribution. It is even missing from OpenBSD base (just confirmed, and loaded it with pkg_add - there are two versions, one with iconv).
So let's take Microsoft's implementation, which will forever lack rsync (as they haven't touched it in well over a year).
The file transfer agent needs to be under the same license.
When you run rsync over ssh, the local rsync starts the local ssh client, which connects to the ssh server, which runs the remote rsync, while the data is piped through the ssh connection.
rsync is available for most operating systems, but it is not usually installed by default. OpenBSD might be the only exception without a rsync package, but it has openrsync. I assume that it uses the same protocol so rsync to/from OpenBSD should work as well as to/from Linux, Windows, MacOS or any *BSD.
If you have the right to transfer files to a computer then normally it is very simple to install rsync, only for your own user if do not have admin rights.
In most cases, you can initially just copy the rsync executable file to the target computer using scp, and then do any other file transfers with rsync over ssh.
After I have abandoned both scp and sftp many years ago, I no longer had any problems with the performance or data integrity of file transfers over ssh.
Nor will OpenBSD.
The rationale for this decision is not technical, it is idealogical.
A technical solution to this problem is insufficient.
To be honest, I was unaware of this:
For maximum utility, it should be bundled with OpenSSH.
Because openrsync has a BSD license, it can be used on any other system that completely avoids GPL programs.