Comment by laumars | Hacker News Reader

Comment by laumars | Hacker News Reader

Parent Full thread

Unfortunately neither of those two statements are true.

1) FTP predates widespread usage of compressed file formats by quite a long time. JPEG, MPEG formats (MP3, AVI, etc), ZIP archived XML documents formats (OOXML, ODF, etc), they're all comparatively new in relation to FTP. GIF is much older than the above but still more than a decade older than FTP and frankly GIF wasn't even that widely used pre-WWW where as uncompressed bitmaps like BMP and PCX were common place. However a lot of text files were copied around in the early days of FTP (hence why FTP has an ASCII transfer mode in addition to binary) and given the modem speeds of the era, compression would have really helped - albeit it's questionable whether the machines were powerful enough to do real time deflating even against text files. That isn't so much a problem these days so it's well worth having compression built into the protocol if just to catch that extra 5%. Even if you think you don't care as an end user, it matters to us sysadmins hosting high traffic servers :)

2) TCP does have a CRC-32 checksum but that doesn't help against failed downloads, memory or other hardware errors etc. A lot of more modern protocols will compare a source and destination hash of each chunk of data and/or the completed file to ensure that the two copies are identical. Frankly, I wouldn't trust FTP for archiving data for that reason alone.

FTP does indeed predate compress (ca. 1984), but I'm unsure of the penetration of IP at the time. UUCP networking was more common and that really only carried 7-bit US ASCII. (Including quite a lot of gifs, I assure you. :-) ) And, yes, on the fly compression would have killed most workstations at the time. My own experience, though, from the late '80s and '90s, was primarily using FTP to get .tar.Z files.

For failed downloads, memory errors, and so on, a hash in the protocol isn't that helpful. You really do want a whole- filled hash check after the file is written persistently. (TCP's checksum is a performance optimization, although it doesn't work very well with wireless networks.)

P.s. "wools"? Stupid android keyboard.

I was on about checksumming the file on disk rather than data in transit. That's how rsync works

I'd forgotten about FTP predating TCP though. Good point.

The really cool modems of the 1988-1992 period implemented proprietary on-the-fly compression on top of 2400 baud. This was standardized as V.42bis.

As a reference, I could usually read a text file faster than 2400 baud (roughly 240 characters per second) but not faster than 2400 baud with V.42bis compression.

The proprietary predecessor was MNP5 (Microcom Network Protocol). The cool thing about MNP5 was that you could implement it on the host PC. There was a DOS terminal program and a FOSSIL driver which worked with my ancient 2400 baud modem (so old it didn't do AT commands) that made BBSing more tolerable.

"TCP does have a CRC-32 checksum"

Where?

TCP has a checksum in the header, but it is neither 32 bits nor is it a CRC.

Maybe I was thinking of the ethernet frame check instead of the TCP header? 32bit did seem a bit high when I wrote I must admit.

FTP is for static files, why would you do on the fly compression? People used to just transfer precompressed files like tar.gz files or .zip files.

FTP has an ascii transfer mode because operating systems have different line endings and this automatically translates for them.

> FTP is for static files, why would you do on the fly compression? People used to just transfer precompressed files like tar.gz or zip files

Indeed they did. But that was when the majority of users were technicians. These days we need a protocol for layman.

> FTP has an ascii transfer mode because operating systems have different line endings and this automatically translates for them.

I know, I left that part you because it wasn't relevant to my point but I'm glad you raised it as it's another feature that isn't necessary these days because, aside Notepad.exe, every text editor on the planet can handle different line endings. Thus ASCII mode is just another point to break files for inexperienced users (I'm not really happy with the kludge that some clients use either; by using the file extension to guess the correct method of transport)

Reply on news.ycombinator.com