Hacker News Reader

Comment by socalgal2 | Hacker News Reader

It is not a schizophrenic zip file to have inline headers that are not referenced in the TOC. the TOC is the only source of truth in a zip file. It was designed this way in purpose so that you can add new versions of files on your 20 disk zip without having to re-write all 20 disks. Pkzip would read the TOC from disk 20, append your new file to disk 20 (or 21 if there was not enough space) and then write a new TOC at the end that does not reference the old file still in the zip. That is by design. Reading anything other than the files in the TOC is an invalid zip reader

View on HN

While that is true (even if some people really want to stream read zips and will thus trigger tar-style conflicts) that is not what the article is about. The article is really more about software using the "size of central directory" field to find the central directory (a sin Python's standard library apparently commits) versus the "offset of central directory".

A commenter on an other post of that article noted that technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect. In fact zip was designed such that the central directory could be split across multiple disks (and later files), so it was not possible to guarantee a simple backwards jump from the EOCD to the start of CD.

> technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect

Yes, and in fact APPNOTE.TXT expects implementations to deal with discontiguous CD/EOCD records; it actually mandates that some records appear between them. (It's where ZIP64-specific records go, for example.)

The whole thing seems to stem from a belief by the author of this post that Info-ZIP is the canonical implementation. (This is of course wrong. That would be PKZIP.)

If Info-ZIP is exhibiting the behavior described, then the Info-ZIP implementation is flat out wrong.

(This has happened before. See <https://news.ycombinator.com/item?id=27925393> where the author of that post provides pushback to Mark Adler, receives pushback from Adler in response, and then just stops pushing and defers to Adler's reading. Adler is an expert in his domain, and he's made valuable contributions to free software, but his expertise is in compression, and he's not the authority on the spec or the format; Adler didn't create ZIP or write APPNOTE.TXT—Phil Katz did.)

Author here :)

> a belief by the author of this post that Info-ZIP is the canonical implementation

↑ Where did you get this from?

I don't think nowadays there's such a thing as a canonical implementation — ZIP implementation world is too fragmented and implementations are too widespread.

One more note is that the article isn't about who's right or wrong in terms of interpreting APPNOTE.TXT — this is besides the point. The point there is that a parser discrepancy can cause security issues, and the article documents just another discrepancy found in the real world implementations (and there are A LOT of these with regards to the ZIP format).

> this makes sense—technically both the offset of the Central Directory and the size of the Central Directory are redundant when you consider that the End of Central Directory Record should be, well, at the end of the Central Directory. In a proper ZIP file the following equation should be true: (position_of_EoCDR - size_of_CD == offset_of_CD)

This is a pretty clear statement on normativity.

You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".)

> This is a pretty clear statement on normativity.

I disagree – it's a technical remark on redundancy of fields (this is related to my hypothesis in the article that redundancy in formats is the primary cause of parser discrepancy). It doesn't acknowledge InfoZIP being canonical in any way.

> You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".)

Fair, though my point still stands.

This is not what the article is talking about.

It's talking about how the EOCD contains both the size of the central directory and the offset of the start of the central directory, which is redundant. So we end up with some tools honoring the offset, while some subtract the size from the EOCD.

It's true that the other comment is unrelated to the content of the article, but it's not true that the offset and size in the EOCD are redundant (for the reasons given in the earlier sibling comment).

And how bad actors could take advantage of that.

I can think they would test which parser you have using some social engineering and create the zip appropriately, but they would also need to know the accountings parser. So I think they would just do the first step and leave it up to chance.

Reply on news.ycombinator.com