Hacker News Reader

Comment by kalleboo | Hacker News Reader

Does the zip format let you replace an arbitrary file inside an archive without rewriting the whole file? It's not uncommon to see Photoshop files that are 400+ MB, and rewriting the whole file on save would have some pretty poor performance. The other examples of files that use the zip approach are either distribution-only (ipa/apk/jar) or generally don't get very big (word processing).

View on HN

You can easily add new files at the end, then rewrite the central directory header (which is at the end of the file for exactly this kind of reason: not have to rewrite everything to add new files). And because the CDH specifies offsets to files, you can replace the old entry and put the new one there (replace old offset by new offset, done). This does however mean the old entry will still be in the file until you "garbage collect". That's if the new version of the file is bigger than the old one of course, if it's smaller or the same size you can just overwrite the old entry and not touch anything.

One could even imagine padding files in the archive so there's some leeway in growing the compressed files, it'd likely depend on the format expression.

This is however the sort of thing SQLite already does for you.

This is an interesting problem, I wonder if Photoshop or The Gimp actually do something fancy here. There shouldn't be much of a problem replacing files when the files never get bigger and uncompressed ZIPs are used (compressed would make any sense for PNGs and JPGs anyway). Small variations in file size could be compensated with some kind of padding, but it seems to me that we are approaching a point of complexity here, where we should just use another kind of filesystem, one that supports fragmented data.

You can append just fine, and SQLite doesn't let you replace arbitrary content either. If the new content fits within the space of what it is overwriting then all is fine (as with zip), but if there is insufficient space then the new data is written at the end (as with zip). SQLite has a vacuum command which creates a new database only with used content.

Reply on news.ycombinator.com