Hacker News Reader

Frankly, I had not realized scp and sftp were two distinct protocols, despite my heavily using them. I thought sftp was an interactive client, while scp was meant for scripting, and never really bothered to dig further. I now will, thank you.

However, I expect plenty of people to be in the same case.

I was definitely in the same boat. I guess I'll just progressively phase out my use of scp in favor of rsync.

actually, sftp is a bit more scriptable than scp. You can pass sftp a "batchfile" of commands and control per-command whether their success should terminate the batch.

scp wins on brevity of commandline syntax

The script-ability of sftp is pretty awkward, it doesen't conform to how virtually every other tool works in terms of "command + args" in a shell scripting context, instead you need to pass it commands through a pipeline, don't love it.

Would be nice to be able to just type sftp user@domain:/file and have it do the right thing.

Oh, that means you have to open a pipe to the process if you want to control it from another programming language? That would be a bit harder to leverage for automation (though I guess it's just a matter of writing a few helper functions, or you could always use bash and redirections if the language is ill-suited to pipes).

Add lftp on top of sftp and you get extensive scriptability. Working public anonymous demo: [1]

[1] - https://tinyvpn.org/sftp/#lftp

Some servers, and some clients, only support one of them. I’ve run into servers without sftp and subsequently stopped using it, as scp always works.

Also rsync does not exist on windows, for instance git bash has only scp AFAIK.

A while ago I tried making windows' remote differential compression COM library work in a CLI utility like rsync.

https://docs.microsoft.com/en-us/previous-versions/windows/d...

You can install cygwin on windows. This will provide sftp, rsync and lftp which is awesome for doing rsync-like transfers on sftp-only chroot setups.

I’ve been using Linux and UNIX systems professionally for ~20 years and this is the first time I realised they were distinct protocols too.

Don't forget some of your other options, such as piping across ssh commands, which is specifically very useful when piping tar output[1]. Piping tar commands traditionally was useful because it dealt better with some file types than scp did (devices, sockets, named pipes, tar is better overall at dealing with them, or was in the past). The other benefit is that after you realize you're just putting a sequence of commands into place to be run remotely, you start realizing you can do other things prior to the tar command if you need...

Rsync's ssh support is generally good, so might be about equivalent, but it's not always handy.

1: https://www.cyberciti.biz/faq/howto-use-tar-command-through-...

I have actually been able to separate the external sftp-server from sshd, and connect it instead to stunnel. The sftp client binary can also be instructed to use something other than the ssh client to establish connectivity.

A simpler invocation could run it through (unencrypted) inetd/socket activation.

SELINUX really throws fits of hysteria when running this, however.

  $ cat sftp-tls.sh
  #!/bin/sh
  exec nc --ssl server 52345


  $ sftp -S ./sftp-tls.sh bogus
  Connected to bogus.
  sftp> cd /nobody
  sftp> get sftp-tls
  Fetching /nobody/sftp-tls to sftp-tls
  /nobody/sftp-tls       100%   15     8.9KB/s   00:00    
  sftp> quit


  # cat /etc/stunnel/sftp-ssl.conf 
  #GLOBAL####
 
  ;sslVersion =     TLSv1.2
  TIMEOUTidle =     6000
  renegotiation     =     no
           FIPS     =     no
        options     =     NO_SSLv2
        options     =     NO_SSLv3
        options     =     SINGLE_DH_USE
        options     =     SINGLE_ECDH_USE
        options     =     CIPHER_SERVER_PREFERENCE
         syslog     =     yes
          debug     =     debug
        setuid      =     nobody
        setgid      =     nobody
  #     chroot      =     /var/empty/stunnel
 
        libwrap     =     no
        service     =     sftp-ssl
  ; cd /var/empty; mkdir -p stunnel/etc; cd stunnel/etc;
  ; echo 'sftp-ssl: ALL EXCEPT localhost' >> hosts.deny;
  ; chcon -t stunnel_etc_t hosts.deny
 
  ; https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
 
 ciphers=ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:RSA+AESGCM:!aNULL:!MD5:!DSS

          curve     =     secp521r1
 
  #CREDENTIALS####
 
  #      verify     =     4
           cert     =     /etc/stunnel/sftp-ssl.pem
 
  #ROLE####
 
           exec     =     /usr/libexec/openssh/sftp-server
  #     execargs=   sftp-server -d /home/nobody
  #     execargs=   sftp-server -l DEBUG3

Checking out the mitigation for CVE-2019-6111 [1], how does `sftp` help?

     * scp(1): Relating to the above changes to scp(1); the scp protocol
       relies on the remote shell for wildcard expansion, so there is no
       infallible way for the client's wildcard matching to perfectly
       reflect the server's. If there is a difference between client and
       server wildcard expansion, the client may refuse files from the
       server. For this reason, we have provided a new "-T" flag to scp
       that disables these client-side checks at the risk of
       reintroducing the attack described above.

You could just do a remote `ls` and then `sftp` all the files listed to your client - nothing stopping a malicious return from `ls`? Surely it's the risk of running any kind of wild card copy? If the server is compromised then there's no telling what will be returned from such a command?

    The scp protocol is outdated, inflexible and not readily fixed. We
    recommend the use of more modern protocols like sftp and rsync for
    file transfer instead.

The protocol itself might be outdated, but surely one of the best aspects of `scp` comes from its simplicity [2]?

[1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6111

[2] https://github.com/openssh/openssh-portable/blob/master/scp....

scp only appears to have simplicity because it outsources large parts of its functionality (e.g. glob expansion) to the destination host's shell. scp/rcp was a great protocol for 1981 (yes, that's when it was introduced) but not for 2019

And this leads to the question.. OpenBSD is famous for removing unwanted or outdated code. What makes scp a special case? Is it too used to be deprecated and removed?

yeah, pretty much. If someone implemented scp's command-line with sftp underneath then we could start the (slow) deprecation process.

its part of the ssh protocol, getting rid of cruft from network protocols is almost completely impossible unfortunately, because its going to destroy many people's workflows, in a way that they can not fix, guaranteed.

Yes, we all know about xkcd, no need to link guys.

scp isn't part of the ssh protocol. It's a command that runs over it.

Out of curiosity, since I wasn't sure which one in particular throwaway2048 was talking about, here are my closest guesses:

927 - Standards - https://xkcd.com/927/

1127 - Workflow - https://xkcd.com/1172/

1323 - Protocol - https://xkcd.com/1323/

IMO sftp is underspecified. The 14 IETF drafts that exist for it have all expired and never reached RFC status.

In practice, this means that the OpenSSH implementation is the de facto standard, and that interoperating a non-OpenSSH client with a non-OpenSSH server is a gamble.

It's interesting that they push for sftp. Personally, if scp is vulnerable to certain types of attacks, I'd rather see an entirely new standard and an effort to avoid having it in draft state for decades.

I'm not sure SCP is specified at all, though. I recently wrote a true SCP client for Go (not SFTP, which oddly enough all the other "SCP" clients for Go use). The only documentation I could find at all was a single blog post from a guy who worked at Oracle.

sftp died in IETF draft because they kept tacking on outdated and misguided concepts from the original FTP like ASCII/binary transfer mode and record file types(really guys, when was the last time an operating system that used this was popular, the 1980s?), as well as tons of other completely pointless cruft to an otherwise very clean protocol, so OpenSSH decided to stay at draft version 3, and being as they are the dominant implementation by far, the effort died.

I'm not sure why we need to see an entirely new standard, sftp as implemented by openssh is well conceived, simple, and covers its use case very well.

> I'm not sure why we need to see an entirely new standard, sftp as implemented by openssh is well conceived, simple, and covers its use case very well.

The key to my point here is "as implemented by openssh".

A relevant anecdote: I had to connect from my program to a third party (version 3) SFTP service, IIRC backed by some Oracle software (but don't quote me on that). I had a version 3 client library. The OpenSSH client worked just fine with the remote, and the client library worked just fine with an OpenSSH server. When connecting to the third party, the client gave up.

After some debugging I learned that the third party server didn't include the language tag in its response statuses (as SFTP version 3 software should). OpenSSH was just fine with this and ignored the problem, and the third party software was probably only ever tested with OpenSSH based clients.

It was definitely a bug in the third party service as far as I was concerned, but I think that this is the natural result of relying on behavior "as implemented by ssh" rather than a stable, well defined (and non-expired) standard.

The only good file transfer solution has always been to use rsync over ssh instead of scp or sftp.

I have stopped using scp and sftp many years ago. Besides the fact that sftp can be too slow, both scp and sftp fail to make identical copies of the transferred files because sometimes they can lose some metadata, e.g. extended attributes or high resolution time stamps.

What I am normally using is:

alias scp='rsync --archive --xattrs --acls --progress --rsh="ssh"'

This gets my vote for nicest alias in the thread.

I've become a huge fan lately of using "--info=progress --no-inc-recursive" for rsync. It can result in more memory usage, since it builds the file list ahead of time rather than dynamically, but it provides you with an overall progress report of the entire transfer, rather than per-file progress reports which may be much less useful.

Aaaah I was so happy to switch until I see "The source and destination cannot both be remote."

Using ssh with tar on both ends is my go to for a new box that doesn't have rsync. Long command lines, but it retains metadata. Like:

  ssh remote 'cd /src;tar -cf - .' | (cd /dst;tar -xf -)

Doesn't call you out so well on typos though.

lftp can be handy too, as that can automatically parallelize transfers.

eg, instead of one connection (eg rsync), lftp can transfer each file using n parallel streams.

Very useful with large files, when there's some kind of bandwidth limit per TCP connection.

Too bad there’s no rsync for windows

WSL

Cygwin provides rsync, sftp and lftp for windows.

Cygwin just feels "heavy" when all I want is rsync. Without any package management (that I'm aware of) I can't just install rsync and whatever other dependencies it has, I have to put hunderds? more binaries on my system.

There is also a linux subsystem for windows 10 which I have never used. Perhaps someone can say if that has rsync.

[ edit ] if that and cygwin are too heavy, you could request the samba team create a posix windows build of rsync. They had one long ago, but stopped supporting it. That would be a single binary. I suspect they will push back.

It does.

This is even heavier. It's a full blown Ubuntu install.

After a bit of Googling, it looks a company has complied and packged rsync for Windows at a cost of $19 per license @ https://www.itefix.net/cwrsync

I found a Github repo with an older (2017) version of cwRsync that may be from before it went behind a paywall @ https://github.com/billyc/cwrsync-installer

Check out MobaXterm. It's free as in beer, and seems to wrap up cygwin and x in one console. It's my go to putty replacement.

You can run a local console and rsync directly from your window drives (/drives)

hmm... works fine in my mobaxterm window.

Msys2 has rsync? And windows has ssh/sshd?

https://docs.microsoft.com/en-us/windows-server/administrati...

https://heejune.me/2018/08/02/setup-rsync-server-over-ssh-on...

Yeah, sftp can't see extended attributes. It's flags are also more cumbersome then scp. For example:

sftp -o IdentityFile=keyfile or scp -i keyfile

Unfortunately, you can't alias scp=sftp as the commandline syntax is different. I'd love someone to write a scp replacement that attempted the sftp protocol first and optionally fell back to scp if that didn't work.

This. I've been using rsync for a long time just for the resume feature even without knowing about some of the other benefits. I tend to use the flags "-avzpP"

Not all ssh servers enable the sftp subsystem by default.

> I think it's the time to "alias scp=sftp". If the developers officially believe that scp should be retired, let's do the switch. Both are parts of OpenSSH and the commandline argument is almost identical.

I've never used sftp before. Does it offer equivalents to:

  scp server:'*.pdf' .

or

  scp server:'*(oc[1])' .

for getting the last created file with zsh extended globs, or

  scp server:'$(custom_remote_program)' .

where custom_remote_program is shell code that outputs a list of files using server-side state?

EDIT: Reading:

> * scp(1): Relating to the above changes to scp(1); the scp protocol relies on the remote shell for wildcard expansion, so there is no infallible way for the client's wildcard matching to perfectly reflect the server's. If there is a difference between client and server wildcard expansion, the client may refuse files from the server. For this reason, we have provided a new "-T" flag to scp that disables these client-side checks at the risk of reintroducing the attack described above.

I guess, I'll actually want:

  alias scp='scp -T'

or just use `-T` on a need by basis. I admit I don't need that feature every week, but it is a relief that they didn't remove the functionality outright. I think the closest equivalent is something like:

  ssh server 'tar c $(custom_remote_program)' | tar xC destination

which I'd rather not be writing.

> I think it's the time to "alias scp=sftp"

May I suggest using rsync instead? The syntax is much closer to scp than sftp is, to the point of being compatible for the most trivial use cases. But rsync is actively developed and has much saner defaults, can continue broken transfers etc.

PuTTY's pscp has a command-line UI like scp, but (by default) it is implemented using the sftp protocol. It's a pity OpenSSH scp doesn't work the same way.

This works pretty well:

alias scp="rsync --partial --progress --rsh=ssh"

You should keep in mind that it is rsync in the background, because parameter are different.

I'd recommend to use `--partial-dir=.rsync-partial` instead of `--partial`, as the latter does not prevent an incomplete transfer from looking like a complete one.