Comment by chasil | Hacker News Reader

Comment by chasil | Hacker News Reader

Parent Full thread

SSH is problematic for several use cases.

It is easy to future-proof the crypto for anything other than a quantum computer:

    Ciphers chacha20-poly1305@openssh.com
    KexAlgorithms curve25519-sha256@libssh.org

That is also the fastest cipher for any hardware lacking AES acceleration, but clients configured this way will never talk to older servers (or vice versa).

File transfer is a real problem for SSH. The scp command is being slowly excised because of its security problems.

https://lwn.net/Articles/835962/

The replacement sftp protocol simply does not perform well, and the performance problems are explicitly not a development priority.

https://daniel.haxx.se/blog/2010/12/08/making-sftp-transfers...

Redesigns are on the table:

https://www.psc.edu/hpn-ssh-home

At this point, if someone baked the chacha20 cipher into vsftpd (replacing TLS), I would take it for internal use.

Wireguard is also an option, allowing us to simply use telnet and cleartext ftp (promiscuous mode on localhost would be a potential path to abuse).

In any case, people where I work have standardized on scp and sftp for communication between many disparate architectures, and few realized the problems this would bring.

rsync over ssh works perfectly (i.e. with the rsync option --rsh="ssh").

Not only it works much faster than scp or sftp, but it also does not lose any file metadata.

I have not verified if newer ssh versions have corrected this, but a few years ago scp and sftp failed to copy extended attributes and on some file systems they truncated timestamps.

I'm probably living in my own little bubble, but I didn't know anyone used sftp for ANYTHING, at least not in the last decade.

Looking at my terminal history, scp for a single file, rsync (which you don't need to pass that --rsh flag to.. during the 21st century ;P) for anything that requires the ability to be interrupted and resume gracefully (which is most cases), and, in the rare case when I need speed above all, the "tar lzop ssh bash pipe" trick (I don't know if it has a canonical name) is correct.

Isn't this what we all do?!? If there's a better way, I don't see how, but would like to know.

I use sftp every now and then I need to pull something and I don't know where exactly that is. A shortcut for finding the path via ssh and then pulling the file via scp, if you will.

That makes sense!

In that rare situation, I typically use tab completion to find the remote file. However, I bet that would SUCK on a high latency or low bandwidth link, where sftp would be just fine.

FWIW sftp vs scp is more about the protocol. Apparently new OpenSSH has an “scp” command that uses the sftp protocol.

But yeah Rsync is a good choice.

Note that sftp != ftps. sftp is scp. Rsync is better if you can get it, but I'm pretty sure that whenever I transfer stuff using WinSCP and File Explorer (Android) it's using sftp.

sftp still used extensively in banking, payments, transferring financial data, or datasets like student information from school districts or information on lawyers from bar associations.

Back in 1999, in one of my first professional gigs, people were still using ftp for sensitive data for automated data transfers.

Rsync+ssh would be great, but enterprise IT is slow to change.

AWS has hosted sftp backed by S3, and it's going to stick around for a while.

Sadly in 2021, ftp is still used extensively in financial services for stuff like clearing stock trades. I wish more institutions use sftp/scp for this and stop relying on allowing specific IP addresses for security.

> I didn't know anyone used sftp for ANYTHING, at least not in the last decade.

We're moving customers from FTP to SFTP. Most recent one was a well-known Fortune 500 company, which uses it to exchange some ASCII flat files with our software.

Mostly it's for XML files or PDFs.

It's simple, it works and it's easy for support to poke and prod.

You raise a good point, and I was incorrect. In my original comment I was thinking of any of us, not muggles.

Come to think of it, I once even set it up at a job in 2009. We needed to digest huge amounts of flat files from clients with little tech savvy. I put FTP on a server, put that behind OpenVPN, then sent the customers a pre-configured OpenVPN client installer. Never broke.

Tangentially, all this mention of FTP being used by banks and financial institutions is inspiring some choice nmap scans to run... ;)

Like someone else said sftp it’s huge for B2B and probably is part of how you get paid. *ftp is blunt but nice when dealing with other parties since it interoperates well, everyone has it, and (most of the time) it just works whether you are talking to a Linux box, IBM MQ, or something else altogether.

> sftp for ANYTHING

Solid Explorer (Android app) connects via sftp to Linux machines just fine. While I use Termux and rsync to do backups and bulk transfers, Solid Explorer allows me to rapidly browse remote files.

> rsync over ssh works perfectly

I use rsync+ssh a lot, and it doesn't exactly work perfectly on long-distance 10-100 gigabit networks: ssh has a buffering problem at high bandwidths which the maintainers won't accept patches to fix.

That is a limitation of ssh.

Using scp or sftp would limit the transfer speed to a value several times lower than rsync in any case.

On short distance 10 Gb/s Ethernet I did not see problems, but it does not surprise me that on long-distance links they appear.

In such cases, if full speed is desired, it is likely that ssh must be replaced with either IPsec or a tunnel over TLS.

scp has exactly the same limitations as ssh, it's the same code.

I don't know anyone in the HPC world who uses "IPsec or a tunnel over TLS".

Tried rclone yet? Does it all, parallel and fast.

What back-end do you use? Also on top of that back-end do use type = crypt, directory_name_encryption = true and so on? I tied various options and found rclone were slow for me.

> ssh has a buffering problem at high bandwidths

This is interesting, can this be the reason for why playing 720p video in Firefox run with ssh and X11 forwarding is fine bandwidth-wise (easily 1Gb/s) but it causes responsiveness to mouse/keyboard actions to go into abysmal 5-30s territory?

Any references to the ssh buffering problem out there?

tar over ssh also does well with file metadata.

I understand and sympathize with rsync advocacy, but there are two major problems here.

The first is the license. Because rsync is under the GPL, it will never be bundled in the OpenSSH.com distribution. It is even missing from OpenBSD base (just confirmed, and loaded it with pkg_add - there are two versions, one with iconv).

So let's take Microsoft's implementation, which will forever lack rsync (as they haven't touched it in well over a year).

The file transfer agent needs to be under the same license.

rsync is a different program than ssh, installed as a different package, so it does not matter what license it has. In most cases, ssh will run a bash at the remote end, which also has the GPL license, so it should be obvious that it is irrelevant what license have the programs launched by ssh.

When you run rsync over ssh, the local rsync starts the local ssh client, which connects to the ssh server, which runs the remote rsync, while the data is piped through the ssh connection.

rsync is available for most operating systems, but it is not usually installed by default. OpenBSD might be the only exception without a rsync package, but it has openrsync. I assume that it uses the same protocol so rsync to/from OpenBSD should work as well as to/from Linux, Windows, MacOS or any *BSD.

If you have the right to transfer files to a computer then normally it is very simple to install rsync, only for your own user if do not have admin rights.

In most cases, you can initially just copy the rsync executable file to the target computer using scp, and then do any other file transfers with rsync over ssh.

After I have abandoned both scp and sftp many years ago, I no longer had any problems with the performance or data integrity of file transfers over ssh.

Android will never, ever install rsync into base.

Nor will OpenBSD.

The rationale for this decision is not technical, it is idealogical.

A technical solution to this problem is insufficient.

To be honest, I was unaware of this:

https://www.openrsync.org/

For maximum utility, it should be bundled with OpenSSH.

I have already written that OpenBSD has openrsync, so there are no problems with OpenBSD.

Because openrsync has a BSD license, it can be used on any other system that completely avoids GPL programs.

> if someone baked the chacha20 cipher into vsftpd (replacing TLS), I would take it for internal use.

Um. TLS can do chacha20 just fine if that's what you want. In particular in TLS 1.3 you can insist on only doing TLS_CHACHA20_POLY1305_SHA256 if for some reason you really don't like AES or maybe block ciphers in general.

The overriding goal was simplicity for an embedded solution.

Yes, throwing up an stunnel in front of port 23 enables telnet-ssl, but this is a bit heavy for a small machine.

I actually have an emulated VAX running VMS 7.3, and the C compiler on this machine does not support 64-bit "long long" so AES-CTR is the best that it can do (and chacha20 won't compile here).

In this case, a version of chacha20 for K&R C would be very handy. As you can imagine, sftp for this machine has been a unique challenge.

Sftp is also not that great. To transfer a folder with children you have to create an empty folder of the same name on the server. Only then you can transfer the client folder, but it doesn't go into the server folder of the same name. Perhaps stack overflow was just entirely incorrect, but that's how I've had to do it with limited systems in the past.

You can also use lftp to transfer a directory tree using the sftp protocol:

  $ lftp sftp://hostname
  lftp ~> cd somedir
  lftp ~> mirror .
  lftp ~> quit

$ tar cf - foo bar | ssh user@host tar -C baz -xvf -

high five

It truly is the best, no? My pipelines look a bit different than yours (they usually have `lzop`/`zstd`, `cd`, and occasionally `pv`), but tar truly is the best file copy utility.

I rarely use `cp -a` even when copying locally. `tar` is so much faster.

Then why isn’t cp-a and alias for tar?

Because that's not how you use it. :) But you raise a decent point. I'll write a short script you could alias to cp in most cases, put it somewhere, and edit this comment in a day with a link.

That can bite you if the remote tar allows absolute paths.

I use this locally as well, simply because I know and prefer tar's behavior to that of cp. For example tar cf - . | (cd /some/where/else; tar xvf -). It's possible to do this using cpio as well but it always seemed like cpio needed so many more switches to do the same thing.

rsync has almost the same syntax as scp, and is really what scp should have been. Use it instead.

Reply on news.ycombinator.com