OpenSSH 8.9 | Hacker News Reader

OpenSSH 8.9

openssh.com

109 pointsthrow0101a35 comments

> Legacy scp/rcp performs wildcard expansion of remote filenames (e.g. "scp host:* .") through the remote shell. This has the side effect of requiring double quoting of shell meta-characters in file names included on scp(1) command-lines, otherwise they could be interpreted as shell commands on the remote side.

This is good, but it broke everything (Fedora made this switch a while back). All our scripts were running scp and doing double quoting so we could deal with whitespace and so on. There's no easy way to tell if the version of scp needs the double quoting or not -- you cannot check version numbers because distros switched scp as early as 8.7 -- so we just had to drop support for older scp and hope for the best.

We weren't using remote wildcards in our scripts, but they're a kind of useful feature which now doesn't work.

The solution is to upgrade to rsync.

Another great option is to mount the remote file system with SSHFS.

sshfs/sftp throughput is very limited. scp acts much better here.

rsync would be the fastest of the lot, plus it does file permissions as well.

sftp is often faster since it starts copying immediately and hides latency more effectively and both sftp and scp copy permissions if you ask them to:

    -p      Preserves modification times, access times, and modes from the
            original files transferred.

Wouldn't `rsync --ignore-existing` have similar latency from sftp then? It will be slower than rsync without the option, but not slower than sftp.

rsync accepts pipes which means you can run end to end compression and decompression automatically during the transfer. Processes can also be parallelized. Most of the time when I see someone claiming rsync does not suit their purpose it ends up being that they aren’t confident in using it safely. Most of the time I never use anything else except the -av flag

It's more likely we'll bite the bullet and use sftp directly from libssh. rsync doesn't work for our use case (and nor does sshfs).

SFTP makes sense here, that's why OpenSSH is deprecating scp because SFTP is much better defined.

I actually wrote all my scp-style scripting with SFTP for years, so much so that I think projects I did this for lived their entire lifecycle and are gone (but I don't know anybody who still works at the place I wrote the earlier ones) without running into the problem I feared - it's just not clear what scp means for non-trivial cases.

Hate to be that guy making you defend a situation you know more about than any of us here..

But I incredulously am curious as to why rsync wouldn't work for you.

I can't speak for them but there are two big ones I've hit:

1. rsync likes to build the entire list of operations before doing anything. If you have workloads which requires scanning lots of small files and/or using something like NFS / s3fs, etc. that means a LOT of I/O delays before a single byte of payload is transferred.

2. The checksum algorithm is really cool if you have large files which only change a little but it's relatively expensive and brittle. I've run into multiple cases where naive sftp was faster because we had a high-bandwidth, high-latency network connection.

FWIW,

1. This is no longer true in rsync-3.0.0 (Mar 2008) and later. It does build the list of course, but transfer begins after establishing just a few directories of content. This "incremental" mode is not available if you specify an option that requires the full list to begin, documented under the "--recursive" option.

2. Checksums are not computed by default. If the files match time/size, they will not be transferred or checksummed unless "--checksum" is turned on. All files which are transferred, are compared by checksum when complete but this is not meaningfully expensive since the IO is already done.

Your issues with high-bandwidth, (very) high-latency links make sense. The rsync algo was designed to minimize bytes-sent over the undersea cable to Australia (low-bandwidth, moderate-latency). This still works well for most internet traffic, but not if your latency numbers are way out of balance!

I'm currently using libssh (not libssh2) in a project and I'm quite happy with it. The API is pretty nice and it behaves pretty much exactly like OpenSSH by default.

We already use libssh all over the place, eg in nbdkit & qemu.

To add to this for completeness sake, if the account is sftp-only one can use the lftp client with its mirror subsystem that replicates most of the functionality of rsync.

OpenSSH is promoting their post-quantum key exchange to the default.

It had appeared that we were waiting for NIST. I wonder what changed.

"ssh(1), sshd(8): add the sntrup761x25519-sha512@openssh.com hybrid ECDH/x25519 + Streamlined NTRU Prime post-quantum KEX to the default KEXAlgorithms list (after the ECDH methods but before the prime-group DH ones). The next release of OpenSSH is likely to make this key exchange the default method."

Cloudflare also recently published a bunch of posts about their transition to post-quantum cryptography. Is it that NIST is close to standardization and organizations are starting to experiment, or is it something else?

The NIST target:

2022/2024 Draft Standards Available

https://csrc.nist.gov/Projects/post-quantum-cryptography/wor...

djb suggested that for openssh instead of the tinydns kex, so tinydns switched also:

https://github.com/janmojzis/tinyssh/issues/50

What is even more surprising about this promotion is that NTRU Prime is not a finalist.

https://csrc.nist.gov/Projects/post-quantum-cryptography/rou...

It appears there have been major disagreements in this process.

"Formal complaint regarding 8 June 2021 incident - 2021.06.15, Daniel J. Bernstein

"Executive summary. A week ago Dr. Daniel Apon from NIST publicly accused me of professional misconduct. Specifically, he accused me of initiating private contact with NIST so as to provide false information to NIST regarding the timing of an upcoming announcement relevant to NIST’s ongoing decisions..."

Thanks for pointing to this conflict between Daniel J. Bernstein and NIST. It is interesting.

After reading the complaint and the official comments from NIST, I choose to believe that Daniel Bernstein is the one saying the truth.

Like in all his writings, Bernstein's version of the events is a very detailed list of precise, easily verifiable facts, while the version of the opponents consists only of vague, evasive claims.

Regarding the reason why NTRU Prime has been selected only as an "Alternate Candidate" and not as a "Round 3 Finalist", this has been explained very clearly by NIST.

There is no known weakness in NTRU Prime, but NIST demands additional research results about the possible attacks against NTRU Prime and against the current finalists, before considering the standardization of NTRU Prime.

While there are some plausible arguments why NTRU Prime might be more secure than NTRU and similar methods, there is no certain proof of this, yet.

What NIST demands is very reasonable, but nevertheless this does not seem to be a good justification for the selection of the finalists.

As NIST says, the finalists include 3 algorithms like NTRU, for which it is possible that someone will find in the future an attack which will work against all 3. Also as NIST says, someone might find in the future a different attack which will work against NTRU Prime.

NIST acknowledges that for now they do not have any idea about which of these 2 will happen first, if at all.

A wiser decision would have been to not put all the eggs in one basket and also select NTRU Prime as a finalist, possibly instead of one of the algorithms with a similar construction like NTRU.

This would have stimulated additional research on both kinds of algorithms. Now if someone finds an attack against NTRU-like algorithms, there are fewer alternatives between the finalists.

It appears that the market is going to choose NTRU Prime in precedence of whatever NIST endorses.

OpenSSH is a huge vote of confidence, and it appears that it is rightly placed.

However, when executing a flat-file signature with OpenSSH, the result will not be protected from a quantum attack.

Is there any evidence of quantum attacks or even the hardware capable of performing such attacks?

There is no hardware capable of running Shor's algorithm.

However, the current worry is that a statistical attack will be found that will run on current quantum annealers (DWave).

https://www.bloomberg.com/news/features/2017-06-14/the-machi...

Whoops I meant this article.

https://www.forbes.com/sites/arthurherman/2021/06/07/q-day-i...

Has anyone else ran into cheap hosting providers where scp would work but sftp doesn't? I hope this change will make them reconsider their configuration.

If SCP works but SFTP doesn't, then they are most likely running an outdated version of OpenSSH with insecure algorithms.

OpenSSH's habit of giving their tools the same name as protocols makes this confusing to talk about. SCP (the tool) uses SFTP (the protocol) by default, right?

scp uses a custom stream protocol over stdout/stdin. It's more like a tarpipe, though shares nothing in terms of format with that.

SFTP is essentially RPC for the various I/O syscalls (open, read, write, ...) - it's basically NFS-as-a-user over SSH. Read/write size is limited to 32 kB per request (in OpenSSH / minimum supported size).

scp the tool uses the SCP protocol by default. Recent versions allow you to use sftp instead:

     -s      Use the SFTP protocol for transfers rather than the original scp
             protocol.

Ah, now I know why I was confused. It hasn't happened yet, but sftp will be the default protocol soon.

> A near-future release of OpenSSH will switch scp(1) from using the legacy scp/rcp protocol to using SFTP by default.

https://www.openssh.com/releasenotes.html

No, many administrators, like myself, disable the SFTP protocol, even if they always use the up-to-date OpenSSH.

I also always remove scp because I use only rsync over ssh, which does not have the inherent bugs of scp and sftp and is also much faster.

SFTP requires the subsystem to be enabled, and can be implemented either by an external binary or an internal OpenSSH implementation (some SSH servers do not implement it internally).

SCP is always external, and merely needs to be found in the path. Because it's much more simple to configure, greater ubiquity is not surprising.

scp is effectively:

  cat foo | ssh user@remote 'cat > foo'

That is why all the escaping is necessary.

random data points - recent scp copies here showed about twenty percent connection overhead. HTTPS copies showed very little overhead, but the HTTPS delivery guarantee is not the same. sshfs/sftp throughput not tested but it would be more than twenty percent.

Reply on news.ycombinator.com