Hacker News Reader

Comment by opless | Hacker News Reader

The articles main point seems to be complaining about ambiguity.

> Scan from the back, find the end-of-central-directory-record and then use it to read through the central directory, only looking at things the central directory references.

It was pretty obvious 30 years ago that this was the correct way to unzip. Streaming content wasn't even a thing back then (dialup being a luxury). It was the only way to ensure that a self extractor could work reliably.

View on HN

The problem is not that the reading method should be obvious once you consider that era, but that the reading or writing method is not exactly specified and there can be a considerable variety in implementations.

For example the OP complains about a passing mention (4.3.1):

> Files MAY be added or replaced within a ZIP file, or deleted.

While the author considers this to imply (but not acknowledge) that local file headers and central directory headers may disagree to each other, I'm not even sure because the specification never clearly confirms whether a random data can appear before any local file header. The only thing that has been acknowledged is SFX (4.1.9), and it is not even specified where the "extraction code" can be embedded. Can it be placed between two files? If such thing is indeed possible, shouldn't there be a provision to avoid ambiguity due to the stray data? Say, it could have said that deleting a file in place should mask all occurrences of local file header signature 0x04034b50 so that the vacant space can no longer interpreted as a file at all.

Exactly, some of the quirks are just a result of designing this for these old systems. ZIP needed to work acceptably on machines with less than a megabyte of RAM. The file format contains ambiguity and redundancy yes, but given well behaving software it's the "good enough" of archive formats. Even in the *nix world we didn't get anything better, tar isn't even seekable and has its own idiosyncrasies, especially if you compress it because the compression happens afterwards and is applied to the whole archive.

Tar IS seekable, as it was designed especially for tapes.

The modern usage of it adds compression, which you have to undo first before doing your archive operations ;)

Zip was > tar (and cpio) in that regard because you didn't have that two step process and only had to have enough spare disk space for the extracted file, rather than extracted file + decompressed tar

Tar is seekable in the literal sense, you have to seek around until you find the file you want. You have to look at each header, if it's not the file you want you skip over that one file and read the next header. Because you only know how long that one file is. So you cannot even know where the third file is without having processed the headers of the previous two. It's "read, skip, read, skip, read skip" until you maybe find the file you wanted. ZIP otoh has the central directory index where you can look things up much faster.

Zip still > tar in that regard, since you can list archive contents (and extract individual files) without having to go through the entire thing every time, hence zip being suitable as a local file system (à la opendocument or ooxml).

Zip is terrible as a local files system, unless you consider read only (randomly, and not many files). Writing/append to a file requires some form of garbage organization/compaction - which requires rewrite of the entire CEN. Unlike the regular files/entries, CEN doesn't have a checksum, so corruption there is pretty bad.

Each file being compressed individually means low compression rates, the deflate overhead might be acceptable for slow mediums (although in that case the memory overhead would be non-trivial).

The massive downsize of zip is that each file is compressed individually, which means very poor compression. Overall the compression rate is significantly lower, same with an attempt to decompress it all. It's only useful for random reads of few select files.

Tar -> read, skip, read, till you find what you need, compression doesn't change much, as most stream compression algorithms have a flush method.

> The massive downsize...

I expect you meant "downside", because when talking about file compression, a "massive downsize" sounds like a good thing :-)

indeed, I suck at typing

Also the only way such a file can be editable which was a major design goal of the zip format and which is not really possible with Tar files.

Not sure what you're saying here. Tar supports appending new files and new versions of files to an existing archive without rewriting the entire archive. I don't believe it supports marking files as deleted (not sure why, tbh) but you can add an empty file with the same name for non-tape storage to prevent extracting the original version. It does this by simply appending the new version of the file to the end of the archive, which is understandable as unlike zip files tar (tape archive) was designed explicitly for tape, where there is no random access. Unless you're referring to editing files in-place inside a tar file, which should actually be easier than doing so with a zip file (as long as the new version isn't longer than the old version, and you're not actually storing it on tape).

So there’s three options: index at the beginning, index at the end and no index.

Index at the beginning means it’s easy to look up, but impossible to add anything.

No index, which is tar, means it’s impossible to tell what’s in it without reading the entire file.

Index at the end, which is zip, means you have to do slightly more work to find the index, but then you can read it without reading the rest of the file, and move it further along the file if you want to extend the file and you can edit it or add to it.

And if you want compression tar is just a compressed blob and the only thing you can do is read it from beginning to end. That’s just not very helpful unless the only things you want to do are packages a bunch of things and unpack all of them. A major use case but one met perfectly fine by hundreds of formats.

Well, depending on the definition you could fit ZMODEM as an streaming + compression (RLE) protocol. I used it a lot almost 30 years ago!

As I mentioned, dialup for many was a luxury.

Streaming was a thing if you include tape

Cassettes were mostly used on 8 bit machines ;)

Tape drives were relatively expensive to floppy disks.

Tape drives were used on a lot of early computers, including the hardware that ran early Unices (e.g. 16-bit PDP-11) and which tar/gzip were written for

https://en.wikipedia.org/wiki/IBM_7-track

https://en.wikipedia.org/wiki/9-track_tape

https://en.wikipedia.org/wiki/DECtape

"tape" includes reel-to-reel systems as well as cassette.

And exactly none of them were used on contemporary machines that PKZIP ran on.

Huh. I wasn't aware that cassette tapes were used on the systems that PKZip ran on either.

Pretty sure DOS didn't have tape drive support ordinarily, and I wasn't aware that PKZip was widely ported to anything that wasn't DOS or a clone/derivative?

https://en.wikipedia.org/wiki/IBM_cassette_tape

Sorry, but I guess you had to be there to have that knowledge?

Cool, TIL.

I was kind of there, but I don't feel like this is something I should necessarily have been aware of:

> apart from one diagnostic tape available from IBM, there seems never to have been any software sold on tape, and the interface was not included on the followup PC XT.

Yeah, I don't think I knew anyone with a first-gen IBM PC, rather than an XT or XT clone. Even so, it seems like even for something as niche as the personal computer was pre-1983, this seems like an extra niche bit of lore. I wouldn't be surprised if a good proportion of people who were properly there didn't have that knowledge.

But, sure, be a douche about it if it makes you feel more superior. Whatever.

> An IBM PC with just an external cassette recorder for storage could only use the built-in ROM BASIC as its operating system, which supported cassette operations. IBM PC DOS had no support for cassette tape,

So, are you saying that PKZip supported running on the pre-XT ROM BASIC OS, as well as PC-DOS/MS-DOS?

> But, sure, be a douche about it if it makes you feel more superior. Whatever.

No, but don't try to assert conjecture as facts. "Whatever"

> So, are you saying that PKZip supported running on the pre-XT ROM BASIC OS, as well as PC-DOS/MS-DOS?

Now you're being a douche, no? I said that the hardware had cassette tape access.

The IBM PC, and PCjr both has the hardware to access tape drives. And yes you could access the cassette tape routines, INT15, 0/1/2/3 ( https://stanislavs.org/helppc/int_15.html ) through DOS, though as you allude to, there was little need to do so.

However BASICA could load programs from tape. https://www.youtube.com/watch?v=KTmzUBb924A

There, something else you can learn today. I'll leave it to you to discover exactly what.

> No, but don't try to assert conjecture as facts.

I hedged with "I wasn't aware", "pretty sure", "ordinarily", and ended with a question mark. I don't know how I could have been more guarded in expressing my recollection as non-authoritative. OK, maybe "pretty sure" was a bit on the strong side. sigh Sorry for that.

> I said that the hardware had cassette tape access.

I thought your point was that IBM cassette drives, unlike the tape systems I mentioned, "were used on contemporary machines that PKZIP ran on."

If your intended meaning was something other than "original IBM PCs running ROM BASIC OS so they could access an IBM Cassette tape was a supported platform by PKZIP", then I apologise for not being smart enough to make whatever leaps of logic were required to understand what your actual point was.

And the winner of the mental gymnastics award goes to ...

PK(A)Zip was available on the Amiga in 1990

https://groups.google.com/g/comp.sys.amiga/c/_XvqYUvBwvc/m/c...

Unix support came later, in the form of infozip in August 1992

I mean they were, but maybe sometimes for backups of hard drives? Certainly not as a common medium to exchange software.

Tape drives for backup were used at the time, and are still used. Though I can't blame PKZIP for not caring about that use case

https://en.wikipedia.org/wiki/Digital_Linear_Tape

https://en.wikipedia.org/wiki/Data8

https://en.wikipedia.org/wiki/Linear_Tape-Open

I'm trying to parse your comment.

Are you being pedantic by suggesting cassettes aren't tapes? When colloquially in the UK at least cassettes were called tapes.

Plus both tapes and cassettes are streaming formats.

And other tape formats came along later for the pc market anyway.

Self-extractors are too dangerous to run, except perhaps in a sandbox.

Apart from 100% of windows installers, and curl | bash installers ? ;-)

Remember we're talking about late 80s/early 90s. Self extractors were how you distributed binaries on floppies to people that might not have had PKZIP and couldn't just go and download it.

Reply on news.ycombinator.com