Stop Using FTP
stevenf.com
stevenf.com
Historically, people tried to convince folks to call it FTPS (for FTP over SSL, which is the most popular form of secured FTP), but that seems to have never taken off. Unfortunate, since SFTP means too many incompatible things.
It's stabilized somewhat in recent years, and usually clients will do the right thing--and the most common meaning of SFTP has come to be "FTP encrypted with SSL" (which is actually one of the weaker forms of security for FTP--but it's got some semblance of "standardization", and works in lots of clients and servers, so that makes it superior). And, of course, there is no RFC defining secure FTP, so it's a de facto "standard" which can lead to buggy implementations and a lot of finger pointing--server says client is wrong, client says server is wrong, user says "I give up" and uses FTP, instead.
:)
Seriously, here's the first DESCRIPTION paragraph of man sftp:
sftp is an interactive file transfer program, similar to ftp(1), which performs all operations over an encrypted
ssh(1) transport. It may also use many features of ssh, such as public key authentication and compression.
sftp connects and logs into the specified host, then enters an interactive command mode.If this is not an ironic comment: RTFM is good for you and it's not a good habit to encourage people to ask for "pre-digested" knowledge just because "you" see no value in looking something up.
No, this is absolutely correct. But don't be surprised if someone misinterprets you.
I didn't consider him saying "like using scp" to mean he thinks "it is scp". Rather, I took it to mean, "it uses SSH to copy files, like scp".
But, I can see how one could interpret it that way. It was not my intent.
I'm staying the heck of out of the pedantry business in the future. This has been a ridiculous bundle of threads all spawned just because I had a bee in my bonnet about how the term SFTP is confusingly misused by lots of folks. I don't even use any of these "FTP" protocols for anything (I use rsync or SVN for website data, and scp for one-off file copies--I haven't used an FTP GUI since I owned an Amiga). I only ever touch FTP for testing problems our users report, and then always using lftp on the command line. I've got no dog in this race...
Which brings up what I was wondering, what use cases do SFTP/FTPS/FSTP etc. satisfy that a combination of scp and rsync do not? Maybe better for GUI client and file browser interfaces?
What, exactly, do you think I have backwards? I just tried to explain how wildly diverse the meanings are, and how people use the terms. I don't care what your favorite FTP client means when it uses the term SFTP (bully for you, that you happen to use one that uses the term correctly)...I'm telling you that there is little agreement in the wild about what the term means, and many people use it incorrectly to refer to FTPS (and other kinds of secured FTP).
Googling SFTP is actually very heartening--it takes quite a lot of searching to find anyone using the term incorrectly. This wasn't always the case (and the hearts and minds of the non-technical masses takes a lot longer). We still see at least 50% of our customers saying "SFTP" when they mean "FTPS" (and we don't support any of the proprietary FTP-with-encryption protocols).
I think part of our users confusion comes from the fact that the server that provides SFTP service is called sshd and the server that provides FTPS is called proftpd. Obviously, FTP is right in the name of one of them, so that's where all FTP comes from! System administration is hard.
Since you haven't provided references, I will - look at Adobe Dreamweaver or other web based clients. SFTP is assumed to mean SFTP as implemented by an SSH server.
Traditional FTP uses two ports and it's a bitch to run via SSL - as a result, it's extremely rare.
PS. Moderated the article down for "This is going to sound a little weird at first considering what I do for a living, but I want you to stop using FTP."
It's 2008. You should have stopped using FTP in 1998.
Many users (and a few clients and servers) do use the term incorrectly. I don't like it, and clearly you don't either...I don't see what you're disagreeing with.
And, of course FTP over SSL is not extremely rare...not even close to rare. All of the major FTP servers, and many FTP clients, support it quite easily, and many, many end users insist on it. ProFTPd, possibly the most popular FTP server, can be configured to use it with just a few configuration directives.
I'm not encouraging anyone to use the terms incorrectly or to use the FTPS protocol instead of SFTP. In fact, I've been recommending SSH FTP for years (one could probably dig up mailing lists posts that include me scolding people for using FTP from ten years ago).
I guess I've not been very clear. I apologize for misleading anyone into thinking I intended to praise the misuse of terms. I just wanted to point out that people use the terms incorrectly extremely often, and it's wise to be explicit about the protocol you mean.
> the most common meaning of SFTP has come to be "FTP encrypted with SSL"
I disagree with that. We both think people should use 'SFTP' to correctly mean as implemented by an SSH server, but you think the term has lost meaning. I am disagreeing with that.
What do you use to make the parsers?
Webmin is written in Perl, and it uses a few relatively simple libraries that Jamie has built over the years (Webmin is 11 years old this October) for slurping files into arrays, breaking them into key=values, etc. Thus each module has a lot of code devoted to parsing...some more than others. We've begun working on making this functionality accessible to other Perl code (and eventually other code in general) via APIs, but that's still coming along slowly...and some of it is unfortunately not abstract enough to be as useful as we'd like. For example, using the Postfix module and the Sendmail module doesn't necessarily have the same semantics--starting and stopping the service are almost always the same between services, but anything config file related is very specific and you have to have knowledge of the directives to do anything with them.
Sometimes the parsers use regexes, sometimes array munging with splice/unsplice/push/pop/etc. grep and map play a big role in Webmin code as well, as they make a lot of kinds of parsing a lot more concise. The code can be intimidating at first, as it's really big (~400kLOC), and because Webmin has never (intentionally) broken backward compatibility for module authors (in eleven years!) has a lot of pretty hairy legacy code. Newer modules and those that are more aggressively maintained (Dovecot, Virtualmin and its related modules, Sendmail, Postfix, Apache, etc.) are nicer to read than older ones.
Anyway, I don't think there is any one silver bullet to deal with configuration files on Linux. They're all pretty wildly different, and some have very bizarre quirks and complexity. Apache, for example, can have arbitrary directives and syntax added by modules, and anything that parses the configuration has to be able to at least understand it well enough to skip over those sections and directives.
I wonder about suggesting a common format capable of storing most config file structures as an RFC, then patching those apps to optionally use the format. If you could get Samba, BIND and Postfix the rest could follow...
One could make the case for something like "everybody use ini syntax", which is the Samba config file format, and the default configuration file format for Python's ConfigParser module (and most other languages have good parsers/writers including Perl). I think it has some concept of nested sections, but I'm not sure it could handle the complexity of BIND or Apache without difficulty. An Apache VirtualHost section can have many sections within, including non-unique sections (e.g. multiple Directory sections), and quite a lot of other tricky stuff, and I'm not able to think of an intuitive way to represent that in ini. I wrote an ad-injecting redirecter for Squid in the distant past that used ini format files...and I ended up having many files in a directory, because it needed nested data. My point is that there's a reason the Apache config file is as complex, and different from everything else, as it is...because it would be hard to convince simpler formats to express all of the information concisely.
Pretty much any config file format can be convinced to do just about anything, though--I'm often stunned at the whacked out crazy crap Jamie does with the Webmin configuration files, which are key=value files. He's got Perl code hiding in some of the values, for example--ternary operators and such--which gets evaluated into place as needed (this is much safer than it sounds). So, whatever format someone came up with would end up being extended as needed and that means you'd have to accomodate all of those extensions--but it still might be nicer for the folks building management tools to have at least a basic standard, even if the management tools don't necessarily know what all of the sections/directives mean.
That said, I suspect that if one could convince a standards body to get behind a "standard" config file format, you'd end up with some XML creature that no one would be happy with, not even those of us who have to parse the configuration files and would ostensibly be relieved to only have one file format for everything.
Momentum is also a problem. I don't foresee anyone on the Apache project, thinking, "Hey! Let's add a second config file parser so people can use SCFF (Standard Config File Format) so Apache configuration files will be the same as a couple of other projects!" It'd be more likely that one or two small projects would use it, and nothing much would change...we'd just have one more configuration file format to parse to support those couple of packages.
Jamie has a sort of funny, and deeply pragmatic, attitude about the whole thing: If it works, why break it? He hates it when anything changes, even to get simpler. New directives to old configuration files. Incompatible changes, even if they are simpler to parse. Etc. So, he'd probably be more angered than pleased to learn that everyone is going to change to some "universal" format.
Now that we're talking about it, I'll mention that Postfix has one of the best configuration systems I've ever seen. key=value main.cf, one-line arrays in master.cf, and space-delimited map files for all of the aliases, virtual domains, transport maps, etc. It takes a lot of files, but it's really easy to parse in just about any language, and I find it really easy to use as a human, as well. And, the kicker is that it's three file formats...so a single config file format wouldn't even work for the single Postfix service!
Most enterprises don't allow outbound SSH through their firewalls, in part because SSH is its own slippery slope: if you're allowing SSH, you're allowing people to build VPN tunnels.
The real-world operational problem that FTP solves is upload. There is no download application that HTTP doesn't solve better than FTP. If Apache shipped with a mod_upload with a reasonable configuration system and a simple, workable web interface for specifying what to put where, FTP could be laid to rest.
The bigger problem is, it's just hard to know what else you're allowing once you let someone "only" access SFTP via SSH.
Most of the BigCo's I work with use "Secure File Transfer" software to solve this problem, which are just glorified upload CGI scripts under SSL. That's the best solution, I think.
This is true for HTTP.
- The spec defines no way of setting the modification dates/times of files.
The HTTP RFC defines the Last-Modified header.
- The spec defines no reliable way of determining the string encoding used for filenames.
The HTTP RFC defines several headers related to encoding.
- FTP was designed to be used interactively by a human sitting at a terminal, not by a GUI application working on the human's behalf. The spec doesn't even define the output format that should be used for directory listings.
This problem doesn't translate exactly to HTTP as it isn't used in the same manner. Directory listings are typically sent as HTML and a user clicks a link which all browsers are able to handle trivially. WebDAV adds features similar to FTP, but it is much more standardized.
- The spec defines no way of dealing with file metadata, such as Unix permissions, owners, and groups.
There are HTTP error codes which indicate lack of permission, but there are no RFC-defined headers for these things. I believe WebDAV does define a method to deal with them.
- FTP requires a minimum of two socket connections to transfer a file: the control connection, which is established first, and then data connections which are created and destroyed every time you transfer a single file, or request a directory listing.
HTTP has no such requirement. Keep-alive prevents sockets from being closed and the same connection can be used to get a directory listing and then transfer a file. There are no separate data and control connections.
- FTP is not friendly with firewalls. Because it constantly needs to establish new connections, this has led us to "passive mode" which might as well be black magic as far as most people are concerned.
This is FTP's biggest flaw and this problem absolutely does not exist with HTTP. Once it is encrypted FTP is impossible to handle in a reasonable manner with a firewall.
In active mode FTP creates one connection to a well-known port for the control channel (21). For the data connection the server attempts to establish a connection from a well-known port (20) to a high-numbered port on the client. This obviously doesn't work well with a firewall in front of the client.
Passive mode modifies this arrangement so that the data channel comes from the client to the server. The problem is the ports used. On both ends the port will be a high-numbered. This requires allowing connections from any high-numbered port (withing a large range) to any high-numbered port. Some firewalls understand the FTP protocol and are able to see the port that is to be used; the server gives the client this port with a command in the control connection. The firewall can then allow that specific connection. Once the control connection is encrypted this is impossible.
Now then. How do his arguments apply equally to HTTP?
On the other hand, HTTP is clearly far superior to FTP, especially if you include extensions like WebDAV and the power of custom headers.
I was just pointing out, insecure formats that don't define how to do listings or set mod times or deal with Unix permissions can be quite good. FTP sucks mostly because of the ports/multiple connection issue, as you pointed out.
Why wouldn't we also stop using Apple Finder, FAT, IE6, non-UTF encodings and bunch of other legacy crap? Ohh-ho-ho... so many topics to blog about...
URLs should begin with afp://, at://, file://, ftp://, http://, https://, nfs://, smb://, cifs://, or vnc://
It's easy enough to get a nice SFTP client though (I prefer Transmit. It even supports Amazon S3)
Cyberduck (http://cyberduck.ch/) is also a good sftp/s3 client for OS X.
A good substitute is MacFUSE and the SSH file system. Your server gets mounted as a disk (read and write!) and it's secure through SSH. If you want goodies like preferences there is a product called ExpanDrive that makes it pretty seamless.
Looking at other industries, like health care, people are more content in spending effort/money on "miracle cures" than worrying about taking steps towards preventing these problems from happening.
The exact same thing happens in say, the Automotive industry - a large percentage of consumers dont take active measures towards ensuring their vehicle is in good running order, and are more content with dealing with the consequences.
The same thing seems to happen when talking about computer security. I dont think it's because people dont care about computer security, they're just more ready to accept the consequences rather than taking preventative steps.
It's a real damn shame too.
... or businesses are just more interested in researching products that have obvious profitability. Let's say you spread the gospel of eating healthy foods. Where is your income coming from?
ftp://server.com/report.pdf
It's that simple.
The only use for ftp is to make uploading stuff to public places easier than http. Apart from that it's useless and redundant like telnet.
So I felt like the article was essentially "Pissing into the wind"
I'd also question the speed argument...for large files, SSH can perform compression and make the transfer faster than FTP.
SSH: 2.0MB/s
SSH w/ compression: 2.5MB/s
FTP: 2.8MB/s
While SSH does not approach the speed of FTP it does come close, perhaps close enough to finally ditch FTP altogether in most situations.
http://www.psc.edu/networking/projects/hpn-ssh/
http://www.psc.edu/networking/projects/hpn-ssh/papers/i2mm-h...
The openssh family of tools have had LFS support for many years. I've used scp several times to copy huge files...and our products have scp as a filesystem backup target, which means it deals with archives that are dozens or hundreds of GBs in size on a daily basis.