FTP Must die the technical explanation | Hacker News Reader

FTP Must die the technical explanation

mywiki.wooledge.org

46 pointsaurelien84 comments

As the author of an FTP server called Net::FTPServer, I must say this article is full of crap. All modern clients request binary mode and passive connections by default. All modern servers can restrict the range of port numbers used on the server side. Clients support encrypted control and/or data connections. You can checksum files using de facto standard commands (it's better than unencrypted http in this regard).

Now some benefits of FTP:

- easy uploading of files

- standardized file listings, delete, mkdir, etc

- extensible protocol via "SITE" commands

- wide availability of clients and servers

It's better to compare FTP to WebDAV. After comparing it you may still think FTP should "die" (whatever that means in the open internet where any two people can run any service they want), but at least you'd be making a sound technical decision, which you're not doing by reading this article.

> - standardized file listings, delete, mkdir, etc

Isn't one of the rant's point specifically that these are not standardised (in a de-jure sense)?

> It's better to compare FTP to WebDAV.

SFTP exists, has none of FTP or WebDAV's issues and (as far as I can tell) ticks all your boxes. Even extensions: http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13#sect...

Those commands really are standardized. The problem is that the rant didn't bother to look at any of the more modern RFCs, nor at de facto standards supported across lots of servers and clients.

SFTP is fine too, assuming you have support, which is not available on many embedded platforms, and even on Windows requires downloading extra tools.

I'm hardly claiming that FTP is the pinnacle of human achievement. The original standards have many obvious flaws, but those have long been fixed. What we have now is maybe not elegant, but it works and the problems with it are not any of the things outlined in the article.

The FTP protocol designers understood the need for something like ASCII mode back in 1985. Something the SFTP people had to re-discover in 2005.

Abhay Bhushan, Postel, Reynolds etc did an awesome job!

edit: wrong rfc

Out of curiosity, why did sftp need ascii mode? I'm drawing a blank trying to think of why sftp would need it.

I use it almost daily in a test/configuration environment to move text conf files to Linux machines we are setting up via scripted process. The Linux hosts have ftp disabled and ssh enabled by default.

I use sftp/scp constantly, but I'm not sure how is an ASCII mode needed/important for that.

Do you transfer text files or binary?

Text I do ascii, always have. Is there a reason, no, but that's how I roll.

I run a MUD which has alot of people from all over the world as developers. Most are not technical and do simple thing like create areas for the game etc. Explaining how to convert files from CRLF to LF only has turned out to be impossible.

When they use modern FTP clients which automatically handle ASCII mode it just works. SFTP seems to work for some users. It probably depends on the clients support for ASCII mode..

As the author of a FTP server called warftpd, I agree :)

Another argument in favor of FTP is that the protocol is simple to implement from scratch, well understood and - well it's an established standard. In stead of re-inventing transfer protocols, creative people can cook up new ways to use it.

Why we should kill all well know old technologies? FTP should remain forever and if somebody wish to use it lets allow it. I don't understand all these kind of hype for new technologies. Are you kidding about Dropbox? Drop an old, well know tech by a fancy new protocol with vendor lock-in? Most of the issues listed here are already fixed in modern clients. I am using FTP every day on local LAN and also on internet. Never had any issue with connectivity, or firewalls (this is why passive mode was added) or file corruption. Not to talk about the ascii mode problem (That was a problem only with ancient unix clients).

Where is he/she suggesting replacing FTP with Dropbox?

> (this is why passive mode was added) Did you not read their entire section on the problems with passive mode? I'm glad it works for your use cases, but for many of us, Rsync / SFTP is by far the better option.

I think he might've been referring to this sibling post:

https://news.ycombinator.com/item?id=11252320

> Are you kidding about Dropbox? Drop an old, well know tech by a fancy new protocol with vendor lock-in?

That a bit of a false dichotomy, isn't it? SFTP (for example) is also a well-known, well-established, and open protocol, and suffers from none of the problems the article lists. The choice is not 'FTP or proprietary'.

But it suffers from other problems like compatibility. I never managed to get the filezilla ftp client to connect to a secure IIS ftp server. And visual studio for instance can't deploy to sftp.

Sounds like Microsoft has to up their game, then...

FTP is 40 years old and was designed for an internet where there was no NAT. Let it die already.

No.

TCP is 40 years old and was designed for a world where there was no smartphones. Let it die already.

I've talked to people that earnestly hold this view and want everything to be on top of udp.

Well, there is QUIC and Mosh seems to be working quite well so it is true that UDP may make more sense in certain use cases.

Yeah there are definitely use cases where UDP makes sense and they're far from rare, but that doesn't mean TCP should be thrown out since there are still many cases where a reliable connection based protocol is better.

TCP doesn't really provide the reliability that people assume it does. UDP is often used because it is expected that the connection will be unreliable.

But why. Because that 3 extra MB on your hard disk occupied by an FTP client?

Otherwise IPv6 will give us back NAT free world again.

I am a big fan of backward compatibility and we should always try to avoid reinventing the wheel.

TCP/IP and email was also invented 40+ years ago.

Standards documents are created by people with a need. They then implement the feature and seek feedback, hence the term RFC.

Ideally rather than being negative, how about discussing how it could be improved in the form of an RFC document.

Also the section on comparing an FTP session to a HTTP GET is just misleading. One is an authenticated long running session where as the other is just a fetch with a connection left hanging (resources on the server are tied up).

I recently had a client order SFTP from me, I thought I had misread and asked if they meant FTPS (FTP with SSL) but they insisted that they wanted SFTP.

This was a case where the client was quite informed and knew that they wanted encryption but that it would never work through a firewall so they ordered SFTP.

I was pleased about this but then came other challenges, MITM protection.

My solution was to use DNSSEC with SSHFP records. Unfortunately I can't guarantee that the client software supports it.

Wouldn't SFTP MITM protection just be regular SSH MITM protection?

Yes it's the same thing, but nobody ever rolls out protection for SSH. It's just a bad habit.

When I look at the ecosystem around me I see mostly tech people using SSH, while the case I described was one where ordinary end users wanted to use SFTP as they had been using FTP in the past.

So I suddenly felt a more urgent need for MITM protection.

With techs I guess the responsibility is often shifted onto them to keep track of their known hosts.

Why would you assume they wanted FTPS? SFTP is both more common and very superior. Also, what MITM attacks are you afraid of with SFTP?

You can use regular TLS CA-signed certificates with FTPS, but not with SFTP.

Except yes, you can use CA-signed certificates with SFTP.

You can? And it validates the CA signature? How?

Accordingly: yes, yes, by using appropriate cryptographic algorithm.

I'm not saying one couldn't in theory implement that, but is that actually implemented? Is there any SSH server that accepts an X509 certificate and a client that accepts a database of CA root certs and checks the server cert?

> I'm not saying one couldn't in theory implement that, but is that actually implemented?

I'm not talking about theory here. I'm talking about what OpenSSH can do.

> Is there any SSH server that accepts an X509 certificate and a client that accepts a database of CA root certs and checks the server cert?

Actually, it may not be X.509 per se. I'm not sure if OpenSSH uses precisely X.509 PKI or if it rolls out its own CA implementation, because I don't use this function. I've read somewhere it's the latter case.

> Also, what MITM attacks are you afraid of with SFTP?

SFTP, like SSH, is "trust on first use". If you don't have some out-of-band mechanism in place to verify the server fingerprint, you're going to have a bad time.

While the CA system isn't perfect, rolling out your own CA to clients is easily automated and verifying certificates from that point on happens automatically.

If a client orders SFTP providing off-band fingerprints is a trivial step.

If that client is a single person, sure. If the client is a 500 person organization where half need access to the server, I can see why people would prefer FTPS with a certificate from their internal CA.

You can do SSH certificates that can sign and revoke keys. For some reason, most people assume TLS when you mention the word certificate. Read the CERTIFICATES section in ssh-keygen(1).

True, but you still need to build your own infrastructure to roll out the CA, vs. X.509, which is implicitly handled by all OSes.

"If security of your authentication credentials matters one iota to you, you'll use scp to transfer files, not FTP."

If security maters then use a secure version of the FTP (SFTP or FTPS). FTP has a greater multi-platform support compared to SCP.

I loved the active/passive mode feature of FTP back before getting broadband. I had an FTP client that would put one server in active mode and one server in passive mode and transfer files directly between them with only status messages going down my 9600 baud connection.

Is FTP that heavily used in 2016? The only use I can see for it now that hasn't been replaced by either a better protocol (i.e. BitTorrent) or a service (i.e. Dropbox) is shared web hosting, even then many now support SFTP (and git).

Don't forget WebDAV.

Did WebDAV ever work reliably?

If WebDAV worked as reliably as FTP more people would use it. Which is actually bad since the tech is rather weak in my opinion. :) So good for the internet that it never really "just worked."

Thanks to our friends at cPanel and the hundreds of thousands of web hosts that embrace and evangelize the platform old-school insecure FTP is still popular and strong and often the documented/default way clients are encourage to upload their files.

Why would I use those on the LAN ?

I don't know about you, but I don't trust my LAN and neither should most people. Between crappy routers and infected laptops, we are better off treating it like another branch of the Internet. Even Google has been backing off the "trusted firewalled LAN" model and relying more on edge security.

Can confirm that as recently as 2012 most major banks, certainly in the UK, were using FTP for transferring their overnight batch payment files (some even over PSTN). Some have managed to move to SFTP (and FTPS), but i would say it's still heavily used in banking/payments.

We can't use SFTP internally because of policies around encryption, so a lot of our stuff uses FTP to move data around.

uh, is the policy that encryption isn't allowed? SFTP is encrypted, FTP isn't.

Many security groups in corporations inspect network traffic for malicious activity. Encryption prevents this, so policies or technical measures may be introduced to ban or limit its use.

As a side note, several years ago, I had a friend who worked for a small research company that had a ban on encryption software. Installing or using PGP on a company computer would be cause for termination.

I expect to see these ideas become more mainstream. In future, companies may only allow certain encryption technologies on their networks/nodes (ones which they hold keys for).

> is the policy that encryption isn't allowed?

Where I worked moving encrypted files over plain FTP was preferred by the business clients because they controlled the encryption passphrases, whereas using SFTP would require operations support, as ops own certificates and encrypted service endpoints.

Much easier for a business user to encrypt a file on their computer using PGP and FTP it to the recipient, passing the passphrase out of band, than raise a service request to pick-up and transmit a file dropped onto a specific egress zone.

FTP (and SFTP) is very heavily used in enterprise software, especially when two pieces of software are being integrated.

(S)FTP support has been baked into huge ERP systems like Oracle and SAP for years (I would be very surprised if BitTorrent was even on a list of development features), and when you're working with a company who has a $100mm SAP installation, you transfer files using (S)FTP.

I just finished a large integration with SAP where all of the data was transferred using files over SFTP. We have a modern REST API they could use as well, but that would've added a lot longer time to the integration as it wasn't already baked into SAP (plus, moving large files over HTTP is not ideal).

I use FTP to transfer files from my scanner to my PC. Doing that, I don't have to run any vendor written software on my PC. For me that matters. And for such use cases, the limitations in the FTP protocol is of little or no concern.

It's still the most common way I get sent large files by clients. For up and downloading 250 files @ ~250 MB ftp works very well. Although much of that is due to ftp clients as much as the ftp protocol. If all web browsers came with a easy way to select all the files in a directory and simply up/download then in parallel with automatic restart at failure then http would probably be just as good.

I never had to use FTP at work in my career. Somehow I always got an SSH login.

FTP always was more of a private thing for me, like BitTorrent.

Ftp is still the fastest file transfer protocol , when you invent something better then you can moan otherwise plz go to fail.

... for large files.

Everything unencrypted will always be faster than the encrypted counterpart. Do you also deliberately avoid HTTPS? And is that minuscule performance gain really worth the risk with FTP?

Not convinced https is a simple substitute for ftp. Think of the new asp.net MVC framework. The http root directory is not the root directory of the MVC app, but a sub directory (which by the way is a good design decision, separating executable from static content). So with https you will only have access to a subdirectory of the application you need to deploy. Not sure you will be able to use https to deploy.

ftpes sftp and ftps are alternative solutions but I found them to be extremely incompatible between softwares. I never managed to get filezilla ftp client to connect to a secure IIS ftp server. And Visual Studio can't deploy to secure ftp, etc.

I did not mean to say that HTTPS was a substitute for FTP. You said that FTP was the fastest protocol, and the only reasonable explanation for why FTP could be faster than anything else would be due to the lack of encryption.

Therefore, I asked if you also avoided HTTPS (and only used HTTP), since HTTP clearly is faster than HTTPS.

But if you didn't mean to say that FTP is the fastest due to the lack of encryption, then why do you think FTP is the fastest?

What makes you think so? Nothing in the FTP protocol makes it faster than, say, HTTP, yet FTP requires more network round-trips to initiate each file transfer.

Even SCP easily saturates a GBit line with negligible overhead, where is FTP significantly faster?

Faster for what kind of transfer? Surely for retrieval it won't be noticeably faster than HTTP, and may be significantly slower? FTP has the advantage of no metadata (possibly significantly less overhead on a small file) that's traded for way more round trips, no transparent compression, no persistent connections and limited to no pipelining.

And if you want to push files SFTP, SCP or rsync over ssh will do that just fine. If you don't care about security and you can't saturate your network connection by default you can just use RC4.

To be honest, I'm surprised that anyone still uses that thing in 2016 - we have rsync, bittorrent, and all the HTTP we can eat. What compels anyone to think "I know, let's use the most broken tool I could possibly find"?

remsync for the win. (http://www.fifi.org/cgi-bin/info2www?(remsync)Overview)

You would be even more surprised to know how many web development companies are still using FTP to upload content to websites.

Every time I hire a company to do Wordpress development on Heroku, one of the first questions is always "yes, we understand that you want to use Heroku, but can you give us the FTP password please?"

The main use of FTP these days seems to be cheap web hosting.

I'm not sure why there aren't more hosts who support SCP/SFTP but the vast majority of web hosts seem to only offer FTP access and that's what less technical users (e.g. non-developer Wordpress admins) have come to expect.

Hmm, the rant is just that - a rant. No solutions and many non-arguments such as outdated acronyms in the RFC.

I am with you that there are other solutions, but they are not easy to use for enterprise customers and that is where FTP still thrives. Rsync? Sysadmins maybe know this, but your marketing guy? BitTorrent? That's for illegal stuff and it is blocked on our company network.

You want to send a client (e.g. a newspaper) all your fancy videos and graphics (remember you are the marketing guy) to publish somewhere. The videos are of course 4GB per file.

What is the easiest way to share these files you have sitting in your "Marketing Campaign 2016" folder? Ask your IT department (just going by your suggestions): - HTTP? Our server cannot handle file transfers larger than 2GB and times out after that. Also most clients don't have resuming capabilities. And don't even start with public file sharers our data is private. - BitTorrent? Is blocked on our network. Also the client told us it is blocked by their ISP - rsync? What is that? I'm a windows guy. Also I doubt that the newspaper will know what to do with some weird command line commands...

Just set up a quick FTP server (or use the existing one) and send the link to the client. Everything works. Resuming. Large downloads. Easy to use. Reliable. "Known" interface as the client just sees folders and files. Many clients available or just the web browser.

I am not surprised at all that FTP is not dying, I myself have often advised customers to "just use FTP" instead of whipping up a custom solution. It just works. The cost is low, the benefit high.

Sadly there currently is no run of the mill solution to replace FTP that "just works" for the average joe.

You're transferring private data over an unencrypted protocol that sends passwords in plaintext?

SFTP does everything FTP does, works in every FTP client I've seen this decade, and is even supported by some FTP servers if you need virtual users.

Virtually every current FTP client also supports FTPS, so security is good enough.

Meanwhile, SFTP does not work with browsers and file managers (unlike FTP[S]), so FTPS works "out of the box" unlike SFTP.

As far as I know the web browser can't resume FTP downloads, and if you're going to install a dedicated downloader anyway, you might as well get one that supports HTTP.

And if you can install an FTP server, you can probably drop Mongoose[1] into the directory that has the files (it's a single binary) and double-click it. There are no more steps, it's already serving those files on port 8080.

[1] https://www.cesanta.com/developer/binary

"HTTP? Our server cannot handle file transfers larger than 2GB and times out after that." - So, the argument against a fundamentally broken protocol is that your instance of an implementation of some other protocol is broken? That's not a very strong argument.

As for "FTP just works" - the same way IE6 used to "just work": there are a lot of hacks happening under the hood, along the whole network path, to keep FTP from breaking (don't get me started on FTP and firewalls).

Command line? You mean "press Win, type cmd, Enter, type ftp, Enter." Right? Are you saying there are GUI tools to do that? This changes EEEVERYTHING! (What makes you think FTP is the only protocol with GUI tools? Is it 2000 yet?)

In other words, user-friendly solutions exist. But (and this is the elephant in the room) they are not quite as convenient for your local IT guy as shrugging it off with "just dump it on this unsecured clunker like it's 1995".

You caught the elephant. FTP is established and people know how to use it. And yes, kind of like IE6. I'm not saying that I personally wouldn't want something different (I do), but I know of nothing else as established I can easily give to IT departments. Command line I was referring to rsync. Try to explain all those cryptic arguments to some IT guys or worse a marketing department.

The world changes slow in some areas, especially slow in the enterprise world. That is the world FTP thrives (plus your average joe we hosted).

Well, I give them a preconfigured executable: the window looks similar to Explorer, and there's the remote server's content, and it asks for password at startup - but it's actually WinSCP with pubkey auth. Compressed, resumable, recursive, and reasonably secure (audit ahoy!).

Mac has SFTP support built in, as does Linux - no issues there.

What compels anyone to think "I know, let's use the most broken tool I could possibly find"?

I need to publish a publicly available, read-only directory. Users can browse the directory structure, downloads shouldn't require exotic tools, client software ideally exists in a base system install, and recursive downloads should be easy.

If you have a better idea, I'm all ears.

Honest question: Why wouldn't SFTP work for this use case? Most decent FTP clients I know also do SFTP. Is the encryption overhead non-trivial?

Most dedicated clients do, but not all. Critically, not the built-in FTP clients of major desktop OSes. Windows Explorer only supports FTP, not SFTP, and AFAIK it's the same with OSX' Finder.

How exotic are tools like WinSCP (https://winscp.net/eng/index.php), Filezilla (https://filezilla-project.org/) or Cyberduck (https://cyberduck.io/?l=en) ?

Exotic enough that I always have to tell users to install one, I can't rely on them having any installed. But literally everyone has an FTP(S)-enabled file manager and browser installed.

I've been working with a new set of software that does precisely that.

    1. Publicly available, with the link.
    2. Immutable data, by definition is read-only.
    3. Users can easily view the root directory and every file enclosed
    4. Uses nothing more than a web browser, with no addons. Right now, it does greatly speed up if you use the peer software.

http://ipfs.io - Interplanetary File System.

It does everything you asked for. The current way the network works well for people who don't run the client is that ipfs.io is running an IPFS->web gateway, where you go to http://ipfs.io/ipfs/[hash]

For example, I'm preparing to go to a 3d printing convention where the bandwidth is very limited. So I'm hosting a slew of 3d printer files. If you are running the peer software, click the localhost link.

http://127.0.0.1:8080/ipfs/QmRBWNHKvz4AmTGN4QyJ4bQMd4fb5vfTd...

http://ipfs.io/ipfs/QmRBWNHKvz4AmTGN4QyJ4bQMd4fb5vfTdacjkb6P...

So yes, there is a potential download of some software. What does that get you? It allows you to download at the full speed of the network (bittorrent based) and uses DHT for finding local peers. You can also add files in the network, and share by simply copy/pasting the link to whom you want to have the files. Interestingly, it indeed solves this: https://xkcd.com/949/

Ideally, the IPFS team is working on a javascript-only IPFS node, where the webpage itself is self-hosting the IPFS gateway. That would also free up phone clients and more exotic tyes of computers with only browser support.

We're running FTP servers for two use cases:

• Embedded. TFTP and regular FTP have reasonable high-quality libraries for all kinds of microcontrollers. HTTP doesn't have a standardized directory browsing mode, and rsync/bt/sftp/… aren't available for those µcs.

• File uploading with easily set up sandboxing and virtual accounts. We could use something else here, but so far it was cheaper to keep FTP running than to set up SFTP that a) doesn't require system-wide accounts (couldn't do that because of username clashes), and b) is sandboxed out of the box without me having to double- and triple-check there's no weird corner case where you can get out. And not-(S)FTP solutions seem to lack good, reliable, easy to use desktop clients. We're dealing with tens of thousands of files and dozens of gigabytes here that need to be shared with non-technical users, and work outside VPNs, so CIFS and most hipster startup crap doesn't really cut it.

I don't think the embedded situation is going to change any time soon, but I'd really appreciate a less painful SFTP solution for the latter case.

Reply on news.ycombinator.com