I'd also like to see updates to IMAP, SMTP and FTP to name a few.
This is also why these platforms often use FTPS (FTP over TLS) instead of SFTP (SSH-based) – SFTP doesn't include any support for record-oriented files, only the stream-oriented files used on Unix and Windows.
Or you could just provide a simple API over HTTP, whether an actual REST system or a single endpoint one or two params with well defined inputs that can be accepted (a CGI, basically). Why bother formalizing some standard when the tools to handle this are so ubiquitous (Apache+$LANG on the server, cURL or wget on the client)?
> SFTP doesn't include any support for record-oriented files, only the stream-oriented files used on Unix and Windows.
That's because SFTP isn't really FTP (in the protocol sense) at all. It's just a specialized shell started after an SSH session/tunnel is created.
That it includes FTP in the name is really just marketing because they wanted to supersede the real FTP. In that respect, it makes sense for them to just cover what 99% of the users of FTP needed and stop.
Yes, I was assuming a more traditional client, which is probably a mistake on my part.
> What possible advantage could one get by replacing it with something hacked together with Apache/curl/wget?
Well, there's always the obvious one, which is firewalls are much easier to deal with, since there aren't two separate ports in use, so you get rid of a whole class of network and firewall errors that are quite common.
If possible, one should use extended passive mode (EPSV) over TLS. Then, we are just talking about two ports instead of one, without any of the connection tracking complexity on middle-boxes that active mode or non-extended passive mode can require (such as rewriting the IP address in the PASV command response to implement NAT). And then, you have to wonder, if you have to substantially change the software in use at the client and server (and possibly even write custom code, per your Apache+$LANG suggestion), are those significant changes really worth it just to save on one extra port open on the firewall?
Oh, sorry, I didn't make it clear before. I'm fully admitting that in the case of record-oriented file requirements, especially on the client side, FTPS probably doesn't have a better solution available.
I was just noting that a vastly simplified firewall configuration that both doesn't require multiple ports opened (or to somehow read the PORT/PASV or EPRT/EPSV commands and open ports dynamically) is simpler, which is an advantage. It doesn't necessarily make up to the record-oriented storage problems that would have to be dealt with in those cases though.
Now, I'm not familiar enough with either of those in such detail to be able to say which one is more complicated, but I have a suspicion.
SFTP also provides what is in effect a shell (or, charitably, a remote filesystem), baked into the protocol. For example, SFTP has SETSTAT, which lets you set file attributes. None of what SFTP allows you to do is particularly difficult in HTTP (some of them are made harder by browsers, but that argument doesn't cary any weight since SFTP already requires you to use something other than a browser), but at the same time, none of them need to be baked into a file transfer protocol, either.
But designing a HTTP API to implement the function that FTP gives you sounds like madness.
In your (Matasano/Latacora) experience, does SFTP register as a security risk / exposure, or does it check the box as long as they aren't using FTP/FTPS?
I'm mostly thinking of Erlang's `ssh_sftpd` application here, where your application can "drive" it by exposing a "file server" for it to talk to as if it was the filesystem.
Sometimes security doesn't matter, though.
Picture a "web-app" like an APT package repo host. (I know the reference impls of ALT hosts are essentially static-site-generators, but ignore that, and imagine an impl that's a web-app with dynamic business logic to expose the relevant routes.)
Such a web-app can be served over plain HTTP. Not even an MITM can exploit an APT client, since it looks for specific files on the server and then checks those files' signatures against a local GPG database.
What would be wrong with having this web-app also be an "SFTP-app"? Or an "FTP-app"? Or even a "telnet-app"?
In all these cases, it'd just be a virtual filesystem being exposed over a wire protocol where the security of the wire protocol is immaterial.
That's more in line with what I was picturing, here. You know how (public) S3 files can be retrieved over HTTP, but also over BitTorrent? Picture adding SFTP to that, as another option. That kind of API.
---
For the situations where the service would benefit from security, though, you can still just obviate the need for relying on SSH's security specifically.
For example, you can expose your SSH server only over a VPN. (I know a lot of people do this, though I personally find it to be more of a hassle than just correctly configuring SSH.)
Or you can expose your SSH server behind an stunnel instance, thereby reducing your configuration work down to just "setting up a TLS-terminating load-balancer", that can then be shared between HTTPS and SSH traffic.
---
Or, if it's just that you don't trust the Erlang SSH impl, you could just hack up OpenSSH to act as an "SSH terminator", forwarding the SSH-channel flows out of each SSH connection, as regular (auth-headered) TCP connections to your backend. Sort of like an fCGI server for SSH. I'm pretty sure the only reason nobody has done this yet is that no language (other than Erlang) has batteries-included support for speaking the server-side of SFTP.
Our SFTP server is written in go and exposes a VFS that has nothing to do with anything on the local file system. It doesn't even allow you to download the files you just uploaded in the same session.
Of course things like the recent libssh CVEs should make anyone exposing non-standard ssh services to the internet a bit nervous.
The biggest problem with FTP[S] is that it interacts poorly with NAT. So maybe the problem is IPv4 and the solution is IPv6.
It also interacts way too well with the NSA and ISP eavesdropping.
Passive mode FTP works fine with firewalls that allow outgoing connections. Even active mode can work using something like Port Control Protocol.
It's only a problem when the network is tightly locked down. And when everyone's answer to that is to use TLS/443 for everything, how is that better than just leaving the unprivileged outgoing ports open? You end up allowing everything either way, but with the everything-over-port-443 outcome you can't even selectively block things anymore.
> It also interacts way too well with the NSA and ISP eavesdropping.
You could say the same about HTTP, which is why they both support TLS.
- Marcus Ranum
I would really like to see each of these disappear to be replaced by new ways of doing things. The whole concept of e-mail needs a new thought. FTP should have already gone the way of the dodo if for nothing else the firewall issues its has.
FTP I agree - SFTP is an existing better option.
It may look like it's working from users' point of view but check how encoding is handled, it's a mess and then why is there still no widely deployed end to end encryption for such a core protocol?
Also DNS and domain is broken too. There's no point using UDP which only brings security problems and still no decent way to encrypt which domain you're querying through your browser and see how domain is governed by a structure which make crappy rules like who owns new TLD and you get some absurd "base" price for each domains instead of something more transparent and public. I wish if you write a book, you could easily have 'some-title.book' or when releasing a movie, you could get 'some-title.movie' but somehow it went wrong and no one can fix it.
Lack of improvements on the security on these points make me wonder if agencies just want to keep them dated so they can tap into people's communication easily instead of making the world a better place where everything is p2p encrypted.
For hiding hostnames eSNI is under development. Cloudflare with a recent Firefox nightly lets you see this for yourself.
(That said, DoH solves a very important, but different problem that is: I want to provide a service with authenticity assurances, but DNSSEC is broken for the users, what can I do? Not many people seem interested on providing that kind of service.)
(Email could use a better binary encoding for attachments than base64, though, since transports are basically 100% 8-bit safe, even if not binary safe. Usenet went with yEnc, which IETF balked at in what is a case of perfect being an enemy of the good).
I have seen plenty of web sites broken by it too, and it's a problem when moving files between Linux and Windows.
(And yes, things would be better if people standardized on binarybody instead of 7bitmime. unfortunately, the Microsoft server announces its support everywhere, but it's broken, so nobody can rely on that one extension (an inside parenthesis, there is a work-around that works everywhere, but it goes against the standard).)
Perfectionism & cert vendors / CA mafia.
I have been playing with workflows lately (in relation to agents) and wonder if that would be an e-mail replacement.
Ftp is surprisingly common when integrating between parties, and this can be expensive to change for non technical reasons.
OTOH, AFAICT, SFTP has largely replaced FTP for the system-to-system role (though ad hoc HTTP-based protocols are also common.)
When users found out about all the issues, they preferred sticking to the old FTP, having a desktop sync app (eg. Google Drive), or using a WebUI for managing files (eg. Dropbox's website) etc.
The problem isn't that there are no standardized way of doing it. The problem is that the existing standardized way of doing it has no adoption, because it's not really optimal for current needs.
The "sandboxing" you're referring to is a serverside chroot, for what it's worth. And, of course, web servers have been doing that since NCSA httpd.
It was also possible for the client and server to fork off data handling processes for the data channel and continue to use the control connection without needing async poll/select loops.
Anyway, it wasn't insane. At the time anyway.
There is a well-established and widely-support IETF standard mechanism (consisting of several IETF standards) for that; it's called WebDAV.
> S3 is probably the closest thing to replacing FTP,
SFTP is much closer to replacing FTP, to the point that people often say FTP when they mean SFTP, which underlies most non-HTTPS enterprise integrations I've seen in the last decade. But SFTP isn't FTP (not even FTP-over-TLS, which is FTPS, which has much less use.)
> Ftp is surprisingly common when integrating between parties
SFTP is very common. FTP (including FTPS) is surprisingly common in the sense that any use of it is surprising given the well-established, battle tested, and superior in every way alternatives that are readily available.
No it doesn't. Don't fix what's not broke, and doubly so if the only motivation for the change is "well, Megacorp (c) says it's the bee's knees".
Saying that anything has no drawbacks is pretty much always going too far.
Yeah, the world's communication before the Internet existed wasn't exactly broken either, so let's not invent anything? Poor choice.