Though maybe it requires way more future vision and planning then I expect.
I'd also like to see updates to IMAP, SMTP and FTP to name a few.
Ftp is surprisingly common when integrating between parties, and this can be expensive to change for non technical reasons.
OTOH, AFAICT, SFTP has largely replaced FTP for the system-to-system role (though ad hoc HTTP-based protocols are also common.)
When users found out about all the issues, they preferred sticking to the old FTP, having a desktop sync app (eg. Google Drive), or using a WebUI for managing files (eg. Dropbox's website) etc.
The problem isn't that there are no standardized way of doing it. The problem is that the existing standardized way of doing it has no adoption, because it's not really optimal for current needs.
The "sandboxing" you're referring to is a serverside chroot, for what it's worth. And, of course, web servers have been doing that since NCSA httpd.
It was also possible for the client and server to fork off data handling processes for the data channel and continue to use the control connection without needing async poll/select loops.
Anyway, it wasn't insane. At the time anyway.
There is a well-established and widely-support IETF standard mechanism (consisting of several IETF standards) for that; it's called WebDAV.
> S3 is probably the closest thing to replacing FTP,
SFTP is much closer to replacing FTP, to the point that people often say FTP when they mean SFTP, which underlies most non-HTTPS enterprise integrations I've seen in the last decade. But SFTP isn't FTP (not even FTP-over-TLS, which is FTPS, which has much less use.)
> Ftp is surprisingly common when integrating between parties
SFTP is very common. FTP (including FTPS) is surprisingly common in the sense that any use of it is surprising given the well-established, battle tested, and superior in every way alternatives that are readily available.
No it doesn't. Don't fix what's not broke, and doubly so if the only motivation for the change is "well, Megacorp (c) says it's the bee's knees".
Saying that anything has no drawbacks is pretty much always going too far.
Yeah, the world's communication before the Internet existed wasn't exactly broken either, so let's not invent anything? Poor choice.
This is also why these platforms often use FTPS (FTP over TLS) instead of SFTP (SSH-based) – SFTP doesn't include any support for record-oriented files, only the stream-oriented files used on Unix and Windows.
Or you could just provide a simple API over HTTP, whether an actual REST system or a single endpoint one or two params with well defined inputs that can be accepted (a CGI, basically). Why bother formalizing some standard when the tools to handle this are so ubiquitous (Apache+$LANG on the server, cURL or wget on the client)?
> SFTP doesn't include any support for record-oriented files, only the stream-oriented files used on Unix and Windows.
That's because SFTP isn't really FTP (in the protocol sense) at all. It's just a specialized shell started after an SSH session/tunnel is created.
That it includes FTP in the name is really just marketing because they wanted to supersede the real FTP. In that respect, it makes sense for them to just cover what 99% of the users of FTP needed and stop.
Yes, I was assuming a more traditional client, which is probably a mistake on my part.
> What possible advantage could one get by replacing it with something hacked together with Apache/curl/wget?
Well, there's always the obvious one, which is firewalls are much easier to deal with, since there aren't two separate ports in use, so you get rid of a whole class of network and firewall errors that are quite common.
If possible, one should use extended passive mode (EPSV) over TLS. Then, we are just talking about two ports instead of one, without any of the connection tracking complexity on middle-boxes that active mode or non-extended passive mode can require (such as rewriting the IP address in the PASV command response to implement NAT). And then, you have to wonder, if you have to substantially change the software in use at the client and server (and possibly even write custom code, per your Apache+$LANG suggestion), are those significant changes really worth it just to save on one extra port open on the firewall?
Oh, sorry, I didn't make it clear before. I'm fully admitting that in the case of record-oriented file requirements, especially on the client side, FTPS probably doesn't have a better solution available.
I was just noting that a vastly simplified firewall configuration that both doesn't require multiple ports opened (or to somehow read the PORT/PASV or EPRT/EPSV commands and open ports dynamically) is simpler, which is an advantage. It doesn't necessarily make up to the record-oriented storage problems that would have to be dealt with in those cases though.
Now, I'm not familiar enough with either of those in such detail to be able to say which one is more complicated, but I have a suspicion.
SFTP also provides what is in effect a shell (or, charitably, a remote filesystem), baked into the protocol. For example, SFTP has SETSTAT, which lets you set file attributes. None of what SFTP allows you to do is particularly difficult in HTTP (some of them are made harder by browsers, but that argument doesn't cary any weight since SFTP already requires you to use something other than a browser), but at the same time, none of them need to be baked into a file transfer protocol, either.
But designing a HTTP API to implement the function that FTP gives you sounds like madness.
In your (Matasano/Latacora) experience, does SFTP register as a security risk / exposure, or does it check the box as long as they aren't using FTP/FTPS?
I'm mostly thinking of Erlang's `ssh_sftpd` application here, where your application can "drive" it by exposing a "file server" for it to talk to as if it was the filesystem.
Sometimes security doesn't matter, though.
Picture a "web-app" like an APT package repo host. (I know the reference impls of ALT hosts are essentially static-site-generators, but ignore that, and imagine an impl that's a web-app with dynamic business logic to expose the relevant routes.)
Such a web-app can be served over plain HTTP. Not even an MITM can exploit an APT client, since it looks for specific files on the server and then checks those files' signatures against a local GPG database.
What would be wrong with having this web-app also be an "SFTP-app"? Or an "FTP-app"? Or even a "telnet-app"?
In all these cases, it'd just be a virtual filesystem being exposed over a wire protocol where the security of the wire protocol is immaterial.
That's more in line with what I was picturing, here. You know how (public) S3 files can be retrieved over HTTP, but also over BitTorrent? Picture adding SFTP to that, as another option. That kind of API.
---
For the situations where the service would benefit from security, though, you can still just obviate the need for relying on SSH's security specifically.
For example, you can expose your SSH server only over a VPN. (I know a lot of people do this, though I personally find it to be more of a hassle than just correctly configuring SSH.)
Or you can expose your SSH server behind an stunnel instance, thereby reducing your configuration work down to just "setting up a TLS-terminating load-balancer", that can then be shared between HTTPS and SSH traffic.
---
Or, if it's just that you don't trust the Erlang SSH impl, you could just hack up OpenSSH to act as an "SSH terminator", forwarding the SSH-channel flows out of each SSH connection, as regular (auth-headered) TCP connections to your backend. Sort of like an fCGI server for SSH. I'm pretty sure the only reason nobody has done this yet is that no language (other than Erlang) has batteries-included support for speaking the server-side of SFTP.
Our SFTP server is written in go and exposes a VFS that has nothing to do with anything on the local file system. It doesn't even allow you to download the files you just uploaded in the same session.
Of course things like the recent libssh CVEs should make anyone exposing non-standard ssh services to the internet a bit nervous.
The biggest problem with FTP[S] is that it interacts poorly with NAT. So maybe the problem is IPv4 and the solution is IPv6.
It also interacts way too well with the NSA and ISP eavesdropping.
Passive mode FTP works fine with firewalls that allow outgoing connections. Even active mode can work using something like Port Control Protocol.
It's only a problem when the network is tightly locked down. And when everyone's answer to that is to use TLS/443 for everything, how is that better than just leaving the unprivileged outgoing ports open? You end up allowing everything either way, but with the everything-over-port-443 outcome you can't even selectively block things anymore.
> It also interacts way too well with the NSA and ISP eavesdropping.
You could say the same about HTTP, which is why they both support TLS.
- Marcus Ranum
I would really like to see each of these disappear to be replaced by new ways of doing things. The whole concept of e-mail needs a new thought. FTP should have already gone the way of the dodo if for nothing else the firewall issues its has.
FTP I agree - SFTP is an existing better option.
It may look like it's working from users' point of view but check how encoding is handled, it's a mess and then why is there still no widely deployed end to end encryption for such a core protocol?
Also DNS and domain is broken too. There's no point using UDP which only brings security problems and still no decent way to encrypt which domain you're querying through your browser and see how domain is governed by a structure which make crappy rules like who owns new TLD and you get some absurd "base" price for each domains instead of something more transparent and public. I wish if you write a book, you could easily have 'some-title.book' or when releasing a movie, you could get 'some-title.movie' but somehow it went wrong and no one can fix it.
Lack of improvements on the security on these points make me wonder if agencies just want to keep them dated so they can tap into people's communication easily instead of making the world a better place where everything is p2p encrypted.
For hiding hostnames eSNI is under development. Cloudflare with a recent Firefox nightly lets you see this for yourself.
(That said, DoH solves a very important, but different problem that is: I want to provide a service with authenticity assurances, but DNSSEC is broken for the users, what can I do? Not many people seem interested on providing that kind of service.)
(Email could use a better binary encoding for attachments than base64, though, since transports are basically 100% 8-bit safe, even if not binary safe. Usenet went with yEnc, which IETF balked at in what is a case of perfect being an enemy of the good).
I have seen plenty of web sites broken by it too, and it's a problem when moving files between Linux and Windows.
(And yes, things would be better if people standardized on binarybody instead of 7bitmime. unfortunately, the Microsoft server announces its support everywhere, but it's broken, so nobody can rely on that one extension (an inside parenthesis, there is a work-around that works everywhere, but it goes against the standard).)
Perfectionism & cert vendors / CA mafia.
I have been playing with workflows lately (in relation to agents) and wonder if that would be an e-mail replacement.
While a lot of the modern web is great, it also has drawbacks that should be looked at very carefully: A simple HTTP page from yonder can still be rendered just fine. A modern page can't be rendered properly a year or so down the line. We need modern protocols to solve modern problems we've created such as huge pages with tons of links and back and forth traffic to many servers. While I think work needs to be done in both directions, I don't know what we really need to go so far as to push down yet another HTTP/x protocol this soon. HTTP/1 is going to be used for quite a while still, HTTP/2 solves a lot, and where it matters (ie: dealing with specialized servers) QUIC is already and has been in use.
Examples?
What I would E.g. agree upon is that http2 is a pain for exotic systems or E.g. microcontrollers, due to the window size and buffering requirements that the protocol imposes. But those systems are still free to stay with HTTP/1.1 for as long as they like. Nobody is going to remove support for it anytime soon.
One can create a non-conforming server in a few lines of code. The actual protocol has a horrendous number of corner cases and weirdness. E.g., determining a message body's size depends on whether content-length is sent, is it chunked, was the request a HEAD request, etc. There's also header folding (whereby a header can be split over multiple lines) though one has the out that the standard allows that to be optional (you can respond w/ 400). Trailers (additional headers after the body) and continuations (allows the client to request a potentially large upload, and the server can reject it prior to the client transmitting the body). Actually parsing headers beyond just a KV mapping, that is, trying to work with any of the actual data in a header is fraught with error: many of them can be specified multiple times, and can thus be in multiple occurrences of the header, or all in a single header but separated by commas, or some mix of the two. (Except cookie headers, those are special.) Some also allow a comma to appear in the value itself, meaning a parser that first splits on comma and then parses the resulting parts is broken.
These text protocols, while "simple" in the sense that a human can generally look at the on-the-wire data and parse with their eyeballs, require complex FSMs to parse correctly. (Complex, relative to an equivalent binary protocol with binary-encoded length-prefixed data or even a text protocol that didn't accommodate every random slightly different way of doing it.) Now, HTTP/2 also adds a lot of functionality above and beyond what HTTP/1 has, so that certainly eats up some of the gains in parsing simplicity (and I think it would be fair to say that the outcome is overall more complex).
I'd much rather have a binary protocol and good tooling. (And that's much more feasible with today's browser's devtools. I would not want to attempt this with the lame excuse for tooling IE 6 had back in the 90s.)
MIME's header encoding of non-ASCII characters is another abomination that I'd toss in this bucket. E.g.,
Subject: =?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=
=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=
Historically, HTTP has allowed field content with text in the
ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
through use of [RFC2047] encoding. In practice, most HTTP header
field values use only a subset of the US-ASCII charset [USASCII].
Newly defined header fields SHOULD limit their field values to
US-ASCII octets. A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.DNS zone files for example are specified incredibly poorly despite their apparent simplicity (multiple optional ambiguous fields, significant whitespace, poorly specified line-folding and paren semantics, DNS record-type dependent record value grammars, vaguely specified length limits, lexical includes, embedded parsing directives, it goes on...). You won't find any 2 implementations that behave the same.
It's typical to talk to engineers who haven't been through serious parser work and get responses like "well, it's just splitting a few strings isn't it?"
As a website operator, the hard part of HTTP/2 was getting TLS certificiates. As an end-user, I'm just happy that popular web pages load faster. And if most of the world still uses HTTP 1.1 I won’t care.
Making a datagram-based protocol available will open up a ton of possibilities for cool realtime apps over HTTP. Currently you’re stuck with WebSockets or WebRTC and for some applications these solutions aren’t good enough.
I can't think of many network applications you couldn't build on top of them.
on http2/spdy, they at least had the decency to publish something before forcing everyone into adoption
A few years ago, web sites and infrastructure had to be built to serve decade-old versions. Now it's reasonable to support only a few years-old versions, and this time frame is continuously shrinking as the current generation of evergreen auto-updating browsers takes more market share.
Though I don't like the bit about forcing adoption onto others, I do think that advancing the market to allow standards changes like this to happen in years instead of decades is a good thing.
they all already have ways to release security patches. firefox has a semi-secret/invisible system addon for that. so much that this is still the current way to release security patches and fix internet-breaking bugs.
rapid release for new features eas invented by microsoft with IE, to push the agenda on standards bodies. it moved so fast that nonody even remember IE5 or 5.5... and thats the reason google have rapid release. nothing to do with security, as proved on the 1st paragraph.