Fault-Tolerant SFTP Scripting | Hacker News Reader

Fault-Tolerant SFTP Scripting

linuxjournal.com

54 pointschasil20 comments

Good scripting examples though in my opinion they left out the most fault tolerant option and that is using the "lftp" client [1] which happens to also speak SFTP and can mimic the behavior of rsync in a chroot sftp-only environment. One of the mirror sub-system options is "--loop" which can repeat the job until there are no differences based on the commands invoked. lftp's mirror subsystem can also spawn multiple TCP connections for a batch of files or even splitting up one large file into multiple streams, thus working around round-trip-time latency.

The downside of lftp's mirror subsystem is that directory enumeration is slower than rsync as there is no corresponding daemon on the other end to assist with the local aspect of remote enumeration. It is just using SFTP commands to do all the work.

[1] - https://linux.die.net/man/1/lftp

I can second this, I have made quite a few scripts using lftp (and also lstftp, and ncftp), and it's been the most reliable option.

> lftp's mirror subsystem is that directory enumeration is slower than rsync as there is no corresponding daemon on the other end

This is a benefit for things like mirroring http hosted file repos, as many do not offer rsync due to security or increased load of running a service. Lftp still does a good job on those.

"Perhaps the PowerShell maintainers might consider retroactively defining $? as an integer type that reports the actual exit status of a completed program. This would be useful in a variety of scenarios."

What I have noticed reading comments online is that people who express satisfaction with or actively promote PowerShell (the name still cracks me up) are often the same people who expressly dislike using UNIX or are blissfully ignorant of the UNIX shell's existence. As such, it's possible they would not miss something like an exit status in $? because they never actually tried using it.

I like $status more than $? :-)

The actual suggestion from the article is the Spanish inverted variant, used to quote a question.

Leave $? as boolean, but add $¿ as integer.

One of the better decisions of microsoft was to say we are just going to use openssh instead of trying to invent their own client, well done.

For the unknowing, use rsync instead of sftp if you possibly can. Odds are high you can.

But if you're dealing with a multi billion dollar company that says "no", then this guide could come in handy.

An untrusted sftp account can be confined to a chroot().

This would be more difficult to do with rsync. I'd also be concerned with the account getting a shell, even if the command= option is set in authorized_keys.

An sftp account is still an ssh account, and you certainly can chroot rsync.

In both cases the client is acessing an ssh server and relying on mere config files to ensure that the ssh server launches something other than a shell.

However rsync is also it's own service that doesn't have to use ssh at all, meaning no possible risk of getting a shell. It can be chrooted both from within and without (it can manage chroots of shares, and it can be chrooted itself). And wire encryption could be layered on some other way. ssh is merely convenient.

SFTP is included in all distributions of OpenSSH, and the procedure to restrict it to chrooted file transfer without shell access is well-known.

It is:

  Match Group sftponly
  ChrootDirectory %h
  ForceCommand internal-sftp 
  AllowTcpForwarding no

(Open)rsync is not included in OpenSSH, and it likely never will be.

There may be multiple approaches to implement the above configuration with either the older GPL or newer BSD implementation of rsync, and the capabilities between the two variants will differ, adding further complexity.

The established best practice is the above configuration, for these reasons.

Like I said, you must always access sftp through the ssh server. It relies on trusting the configs to avoid a shell.

It means it's possible for sshd to supply a shell if it's merely confused about the config. It doesn't matter how "established" the config is. You are looking at what the config is supposed to do and thinking that is all nice and bulletproof looking. But that nice config that says "user X only gets the sftp subsystem" goes right out the window if you merely confuse sshd that you are not "user X" or any of the other match conditions.

Rsyncd on the other hand, has no code in it to offer up a shell. It can't be tricked into doing something that it has no code for in the first place.

The only reason ssh is usually involved with rsync is simply that rsync is not encrypted, so you tunnel rsync over some other channel to add encryption to it, and ssh is the most handy. But that other channel can actually be anything. It doesn't have to be ssh, and it doesn't even have to be any other form of shell server. For instance you could run a direct rsyncd and access it via vpn or tls tunnel, where together the two parts get you encrypted rsync and neither component posesses any ability to launch a shell, no matter what happens with the configs and no mattet what exploits come along.

It is not strictly true that SFTP must be accessed through SSH.

I actually jacked it into stunnel once, full details in that past post, which google remembers quite well:

https://news.ycombinator.com/item?id=19520422

Tinyssh also has no config file; if you want it to service sftp, it must be passed in argv at startup, so less confusion there.

And there are two rsyncs now, so you have to be specific about which one. You likely mean the original, and not this one (the same people who wrote this had a showstopper bug in their new mail server):

https://www.openrsync.org/

That stunnel hack is cool.

Thanks; I guess a had a few idle moments.

Why do you think rsync is a better alternative to SFTP when merely uploading files? rsync is great when syncing large amounts of files, but I see no advantages when needing to have a method for customers to upload files.

We recently had to build an SFTP based file upload system for our customers, complete with generating temporary ssh keypairs etc as they have no other way of sharing data with us that would pass various security audits. If you told me 20 years ago that exchanging data with FTP would still be state of the art now, I’d probably would have changed careers. Literally billions sunk in tech companies and and only some incremental improvements in the fundamental network protocols.

The only things sftp and ftp have in common are three letters and an ability to transfer files.

All the bad stuff about ftp is not present in sftp.

The REGET and REPUT commands are implemented by both, which is distinctive.

The SITE command is thankfully not implemented in sftp, and the commands that traverse the wire are not ASCII, so they are indeed very different.

Sftp also left behind the ASCII/binary mangling of line endings.

I think the "ftp must die" paper reveals many of the foibles of ftp.

https://mywiki.wooledge.org/FtpMustDie

I think of rsync as the fault tolerant version of sftp, yes it operates at a different layer(files vs network streams), but in the face of a network that keeps cutting off just keep rsyncing until it gets through.

Reply on news.ycombinator.com