Hacker News Reader

Comment by 1vuio0pswjnm7 | Hacker News Reader

Funny I was just writing a simple .zip file filter (carver) and thinking about those initials in every .zip file, "PK". The filter takes an HTTP stream (headers, chunking, etc.) on stdin and outputs only the .zip file on stdout.

When studying the zip format, you can find yourself thinking, wow, this is really brilliant and then you remember the sad story of the guy who created it (this article in particular).

View on HN

zip is a horribly designed format

https://games.greggman.com/game/zip-rant/

There were precedents of well designed file formats long before .zip hit the scene. Of course there wasn't the internet so being aware of better format designs might not have been common

It's interesting to compare zip with lzh, which was very popular in Japan; they're both from roughly the same era, yet lzh definitely seems far more logically designed (and often outperformed zip in compression ratio too.)

from my memory, both ARJ and LZH outperformed ZIP for my benchmarks. I used to stripe archives across as many floppy disks as I had until I eventually got a CD writer...

ZIP commonly uses DEFLATE which is LZ with a 32k window + Huffman, whereas LZH went up to LZ with a 64k window + Huffman. I suspect the doubled window alone is responsible for much of the improvement.

According to http://fileformats.archiveteam.org/wiki/ARJ , ARJ uses something similar to LZH's lh6 algorithm, which is LZ with a 32k window + Huffman, so the difference could be due to ARJ's reduced file format overhead compared to ZIP, or perhaps slightly better Huffman codes.

> But, that contradicts 4.1.9 that says zip files maybe be streamed. If zip files can be streamed then both of the example above would fail because in the first case we'd see file B and in the second we'd see file A (old) before we saw that the central directory doesn't reference them. If you have to wait for the central directory before you can correctly use any of the entries then functionally you can not stream zip files.

You cannot filter a particular file in a stream using UNIX fifos and constant memory, but you could replace the old file A with the second file A in the stream because it is later. In this way, yes you can consume a zip file in a streaming way, even if you have to wait until the end to pass any of the output to another process.

TAR and CPIO have the same "feature".

No you can't because the definition of "streaming" is to use the data as it comes down the wire. So

    CMD.BAT  format disk
    CMD.BAT  rem format disk

If the TOC only points to the second CMD.BAT then executing the first one as you stream it is wrong.

> If the TOC only points to the second CMD.BAT then executing the first one as you stream it is wrong.

The output directory contains the second CMD.BAT when you're done processing, whether you use the streaming algorithm or you consult the TOC, because in the streaming algorithm you simply overwrite the second CMD.BAT (if you created it as a result of the first CMD.BAT) with the new contents.

There's no point in "streaming" if you're not using the data as it arrives. That's the entire meaning of "streaming".

What else can you do?

You can't wait for the "central directory" record at the end, because now you've missed everything (you can't stream backwards!)

Computer communications predates the internet. We definitely knew about them. Nobody wanted to pay for them. Also, it was in everyone's best interest to informally standardize on a tool -- in those days, having to download an extraction tool in addition to the archive you wanted was a nontrivial cost (real monetary cost, not just opportunity cost of wasting more time).

Once SEA decided to start a legal fight, consensus was reached pretty quickly that nobody wanted to be the one jerk left using ARC. There were semi-organized grassroots campaigns to get laggards to switch.

The personal computer software world was less stringent about standards at the time, and the tool that worked well enough was always going to beat the tool that pleased the software engineers.

The reference to "arc wars" in the original article seemed interesting. I did a quick search and was pleased to have found these at textfiles.com

http://textfiles.com/computers/sea.txt http://textfiles.com/computers/arcsuit.txt

"SEA claimed trademark infringement on the name "ARC," and violation of their copyright on the "look and feel" of ARC's command-line user- interface, in addition to charging Katz with appropriating ARC program code."

The "look and feel" of the command line interface. By the mid-1990s people seemed to assume this only applied to GUIs.

My comment wasn't about paying for formats. It was about designing formats. The zip format is extremely poorly designed. See linked article above

I already know the zip format. Formats were not open then. You did not select archive tools based on the on-disk format it used; you selected archive tools based on a combination of software cost, compression performance, and network effects. Having multiple implementations of software that could work on the same archive format was a novelty which often resulted in legal battles, as the formats themselves were often considered proprietary information.

You still seem to be arguing over compression formats. I'm not saying there were better or worse compression formats. I'm only saying

(a) zip is a badly designed format ... see article on why

(b) the concept of a well designed file format (related to compression or not) was already well known in 1989 when zip shipped.

One big part about your post is about streaming. I don't think the ZIP format was ever designed for streaming, and there are no claims about streaming in the original app notes. Actually, the wording about streaming you refer to were only added to the app note last year, and as you rightly point out, it's not guaranteed to work.

Out of curiosity, is there any reason you don't mention zip.js at all in the documentation of unzipit?

Here is Phil Katz on the Computer Chronicles TV series:

https://archive.org/download/filecompress

Reply on news.ycombinator.com