Hacker News Reader

Comment by raspyberr | Hacker News Reader

What's actually a convenient way of transferring files across a network if scp is being removed?

Update: Thanks for the replies. I should've done a bit of extra research about scp first. Guess this is an example of Cunningham's Law.

View on HN

The scp protocol is being removed, but the scp command you find convenient isn't going anywhere. So that convenience ought not to change, except that now the quoting does what you probably expect, because it's just a local command and so nothing weird is happening.

I'm somewhat of an old fart, but rsync(1) has worked well for me. It integrates well with ssh on both sides, using ssh channels to execute the binary on the server side and transporting data nicely back and forth.

It isn't suited for millions of files, but neither is scp.

It handles as many files as I've ever thrown at it - often in the millions. The great thing about rsync is it is restart-able and the source and destination don't even have to be local.

the biggest weakness of rsync is that it's single-threaded. Single-thread-copy with millions of small files is painfully slow, Rsync is as good as it can be but you just need more threads.

I think that's what a sibling is getting into with the "better to tar files and send them over ssh in some cases" thing. And yes, you can hack it in after-the-fact with xargs/etc but it's clunky compared to just having native multithreading like rclone/etc.

If you use GNU Parallel, there's a fairly easy way to parallelize [1].

That being said, I hadn't heard of rclone - thanks for mentioning it, it looks amazing. I'll definitely be trying this out for my use cases...

[1] http://www.gnu.org/software/parallel/man.html#example-parall...

Wow, Parallel also looks awesome. Never seen it before.

It handles millions, but it can be a lot faster to just pipe output from tar through the ssh connection.

Got any benchmarks/write ups on the subject? I did a bit of testing myself a long time ago and basically the answer just ended up being to use rsync because any differences were marginal. That said, I didn't test with millions of files.

I think perhaps this was a bigger issue back in the day when we were using rotating harddisks. In those day doing a seek would be a lot slower than doing a write.

Today seeks are mostly instants, so maybe my experience isn't valid anymore.

Your experience is valid today but for a different reason: if you're comparing millions of tiny files there's a lot of back and forth. If you're streaming a single archive, it only checks if that single file has been modified.

Like everyone here I've no benchmarks but have got burned trying to rsync around too many small files.

I got into the « lot of small files » case, and I ended up generating a file list, split it and feed multiple rsync instances with xargs.

We've put rsync in a HPC scheduler and used it, or some tooling ontop of it really, to copy billions of files for a large-ish HPC compute cluster with many P of data.

That's my experience - millions or billions of files - who cares?

rsync is time served. It just works. I'm sure there are other funky solutions but they are not proven over decades.

we have had problems with millions of rsunk files.

Also, rsync can create more faithful copies than scp. E.g. scp can't copy a symlink as a symlink, instead it will follow the symlink and copy the file or directory.

This is one of the main reasons I never use scp and always use rsync -a. IMO, seems a poor design of scp.

A simple unix-like way would be to use tar with plain ssh.

To transfer a set of files from local device to a remote device is as easy as:

    tar -c path/to/file1 path/to/file2 ... | ssh user@hostname tar -x

The flags can be memorized with a mnemonic:

    -c = create (archive file)
    -x = extract (archive file)

This could bite you on platforms where tar observes absolute links (commercial UNIX comes to mind).

There is wild variation in tar implementations beyond what is required by POSIX.

I use "gtar" on the command-line in that case; it either fails because gnu-tar is missing, or it works as expected.

"-H pax" is also an important item to specify on the side creating the tar file, as pax support is much older than pax as a default, and the pre-pax default (gnu or USTAR?) has a short enough maximum path length as to have caused me problems in the past.

Or dd | tar | ssh so you can control the size of compressed data (as in very large files)

https://www.linode.com/docs/guides/copying-a-disk-image-over...

The scp tool continues to exist, but implementation switches to SFTP as protocol from SCP protocol.

They plan to deprecate scp the protocol, not scp the tool, which will use SFTP under the hood.

the only snag about this plan is that the old `scp` tool works with any server, whereas the new tool will require additional configuration on the server to enable SFTP.

It's not enabled by default in some distributions and many "how to secure your server" guides recommend disabling it/keeping it disabled due to the best practice of only enabling necessary services.

This will break `scp` for many corporate cases (where arguing that sftp is necessary is an impossible fight)

The fact scp "worked" illustrates that such "How to secure your server" guides are... at best... poorly thought out. If "remote file access" is not in fact a "necessary service" then you can't very well allow scp to work, and thus you can't allow basic ssh service.

Actually a more sane setup, which I've used, is only SFTP is enabled, no shells. You can fetch files, you can send files, but you can't... for example... run a race exploitation shell script to seize root permissions.

This also has the advantage that if we can only send and receive files, the "server" we're connecting to needn't really exist, like an HTTP virtual host it's just a bunch of "files" and could actually be backed by a database with some blob tables.

There's a lot of scp usage in a typical corporate environment that should actually be files living in a shared filestore anyway, I've seen way too many files which ought to actually be:

* A wiki entry

* Checked into a git repository

* Shared in some cloud service e.g. OneCloud

... but instead they live on Steve's F:\Documents and if Steve is off sick well, here's the copy I made last week, I hope it isn't missing any important updates and somebody make sure Steve gets this when we change it...

The file living in /usr/project/ on testserver03 instead of F:\Documents on Steve's laptop is not a real improvement.

You're actually making the opposite argument that you think you're making: if the scp command stops working, from a "corporate" perspective, from all intents and purposes, it means that "our server no longer works with scp due to an update". Regardless of implementation details, the opposite will then happen: whoever is in charge will request for "SCP to be fixed", which will actually mean turning on SFTP support.

Removing options from "corporate" users usually results in saner choices being made, because you stop being able to bikeshed the implementation details. People have a server and they need a way to transfer files. If you have two protocols that can be used, arbitrary decisions will be made. If the readily available tools all break, there is no more decision to be made – whatever needs to be enabled will be enabled.

if regular ssh is available, nothing stops people from building pipes with tar on both ends. This will be integrated into additional tools (future "show HN - scp replacement written in $FANCY_LANGUAGE that works even when sftp is disabled") and will be the "new way" of doing things because it won't require any configuration change on the server.

Never underestimate the inertia of server configurations.

> (future "show HN - scp replacement written in $FANCY_LANGUAGE that works even when sftp is disabled")

Too late:

https://en.wikipedia.org/wiki/Files_transferred_over_shell_p...

In KDE this is implemented as a standard protocol handler ("ioslave") and you can access it from most KDE apps using URLs like "fish://user@server/":

https://docs.kde.org/trunk5/en/kio-extras/kioslave5/fish/ind...

The snag in tbis plan is that wildcard copies dont work in the new scp tool. Serious usability downgrade to the new system. :(

I'm pretty sure scp isn't a protocol. It is just a tool that relies on doing a request for execution over the regular ssh2 protocol.

I've written a few custom ssh servers and you don't really need to do anything to them to support scp.

sftp uses the sftp subsystem. scp just uses regular ssh requests and transports data over the ssh channels.

See <https://en.wikipedia.org/wiki/Secure_copy_protocol>

SFTP also transports data over SSH channels.

I guess you can call it a (application) protocol. However, when implementing this I've not needed to anything to make scp works as opposed to anything else. Just supporting ssh executions requests and connecting the channels as you do with any remote invocation.

The protocol is being deprecated not the scp command. It's been reimplemented over sftp. They have had to handle wildcard/glob expansion, and how scp tried to deal with it.

TL;DR it's just the same command, mostly, for now.

scp the protocol is being replace by the sftp protocol, scp the tool will use that new protocol

sftp, rsync, scp using the sftp protocol in the background (which is now the default).

scp and sftp make sense, but is that really the case for rsync?

As far as I remember, rsync does a few neat tricks like hashing files on both sides of a connection to determine whether it's worth transferring them over a potentially slow connection, which would not work with pure sftp.

You're correct, rsync operates over SSH (or its own native protocol) but runs a copy of the rsync binary on the remote side rather than using the SFTP subsystem.

Reply on news.ycombinator.com