Hacker News Reader

Comment by rwmj | Hacker News Reader

> Legacy scp/rcp performs wildcard expansion of remote filenames (e.g. "scp host:* .") through the remote shell. This has the side effect of requiring double quoting of shell meta-characters in file names included on scp(1) command-lines, otherwise they could be interpreted as shell commands on the remote side.

This is good, but it broke everything (Fedora made this switch a while back). All our scripts were running scp and doing double quoting so we could deal with whitespace and so on. There's no easy way to tell if the version of scp needs the double quoting or not -- you cannot check version numbers because distros switched scp as early as 8.7 -- so we just had to drop support for older scp and hope for the best.

We weren't using remote wildcards in our scripts, but they're a kind of useful feature which now doesn't work.

View on HN

The solution is to upgrade to rsync.

To add to this for completeness sake, if the account is sftp-only one can use the lftp client with its mirror subsystem that replicates most of the functionality of rsync.

It's more likely we'll bite the bullet and use sftp directly from libssh. rsync doesn't work for our use case (and nor does sshfs).

SFTP makes sense here, that's why OpenSSH is deprecating scp because SFTP is much better defined.

I actually wrote all my scp-style scripting with SFTP for years, so much so that I think projects I did this for lived their entire lifecycle and are gone (but I don't know anybody who still works at the place I wrote the earlier ones) without running into the problem I feared - it's just not clear what scp means for non-trivial cases.

Hate to be that guy making you defend a situation you know more about than any of us here..

But I incredulously am curious as to why rsync wouldn't work for you.

I can't speak for them but there are two big ones I've hit:

1. rsync likes to build the entire list of operations before doing anything. If you have workloads which requires scanning lots of small files and/or using something like NFS / s3fs, etc. that means a LOT of I/O delays before a single byte of payload is transferred.

2. The checksum algorithm is really cool if you have large files which only change a little but it's relatively expensive and brittle. I've run into multiple cases where naive sftp was faster because we had a high-bandwidth, high-latency network connection.

FWIW,

1. This is no longer true in rsync-3.0.0 (Mar 2008) and later. It does build the list of course, but transfer begins after establishing just a few directories of content. This "incremental" mode is not available if you specify an option that requires the full list to begin, documented under the "--recursive" option.

2. Checksums are not computed by default. If the files match time/size, they will not be transferred or checksummed unless "--checksum" is turned on. All files which are transferred, are compared by checksum when complete but this is not meaningfully expensive since the IO is already done.

Your issues with high-bandwidth, (very) high-latency links make sense. The rsync algo was designed to minimize bytes-sent over the undersea cable to Australia (low-bandwidth, moderate-latency). This still works well for most internet traffic, but not if your latency numbers are way out of balance!

I'm currently using libssh (not libssh2) in a project and I'm quite happy with it. The API is pretty nice and it behaves pretty much exactly like OpenSSH by default.

We already use libssh all over the place, eg in nbdkit & qemu.

Another great option is to mount the remote file system with SSHFS.

sshfs/sftp throughput is very limited. scp acts much better here.

rsync would be the fastest of the lot, plus it does file permissions as well.

sftp is often faster since it starts copying immediately and hides latency more effectively and both sftp and scp copy permissions if you ask them to:

    -p      Preserves modification times, access times, and modes from the
            original files transferred.

rsync accepts pipes which means you can run end to end compression and decompression automatically during the transfer. Processes can also be parallelized. Most of the time when I see someone claiming rsync does not suit their purpose it ends up being that they aren’t confident in using it safely. Most of the time I never use anything else except the -av flag

Wouldn't `rsync --ignore-existing` have similar latency from sftp then? It will be slower than rsync without the option, but not slower than sftp.

Reply on news.ycombinator.com