Hacker News Reader

Comment by rogerbinns | Hacker News Reader

FXG could have solved the single file problem by grouping all the files up and making a zip file, and picking a different extension. For example this is how Android (.apk) and iOS (.ipa) applications are distributed.

It also has the advantage that if you extract the zip you can now access whatever pieces are inside as separate files even by tools that don't understand the containing format. (eg a bunch of jpg/png inside the zip are perfectly viewable in any image viewer). Trying to do the same with SQLite is problematic as you have to write queries to grab the blobs and then dump them to disk.

Where SQLite as a format does shine is if you need to keep historical information such as an undo/redo log. That makes it good for an application format, but not for an interchange format.

View on HN

> For example this is how Android (.apk) and iOS (.ipa) applications are distributed.

A more relevant example: this is how .docx files work, for very much the same reasons (one XML file at the root, and then a bunch of referenced resources.)

Apple also uses zip for Keynote, Pages and Numbers documents, again for very much the same reasons, and Java's .jar, .war, .ear, .whatevar files also use zip.

.zip seems to be the universal 'combine some stuff in a single file' format, probably because it is old enough to be guaranteed patent free (OTOH: http://broadcast.oreilly.com/2010/06/is-zip-in-the-public-do... and http://www.pkware.com/support/zip-app-note. http://www.pkware.com/documents/casestudies/APPNOTE.TXT only mentions encryption and patching as patented)

It also, unlike tar.gz, provides an index for easily seeking to and extracting just one file from the zip archive. My understanding is that doing this in a .tar.gz means unzipping and processing the whole tarball as a stream.

Indeed--squashfs was basically created to deal with the inadequacies of random-access-extraction from compressed tarballs.

Does the zip format let you replace an arbitrary file inside an archive without rewriting the whole file? It's not uncommon to see Photoshop files that are 400+ MB, and rewriting the whole file on save would have some pretty poor performance. The other examples of files that use the zip approach are either distribution-only (ipa/apk/jar) or generally don't get very big (word processing).

This is an interesting problem, I wonder if Photoshop or The Gimp actually do something fancy here. There shouldn't be much of a problem replacing files when the files never get bigger and uncompressed ZIPs are used (compressed would make any sense for PNGs and JPGs anyway). Small variations in file size could be compensated with some kind of padding, but it seems to me that we are approaching a point of complexity here, where we should just use another kind of filesystem, one that supports fragmented data.

You can easily add new files at the end, then rewrite the central directory header (which is at the end of the file for exactly this kind of reason: not have to rewrite everything to add new files). And because the CDH specifies offsets to files, you can replace the old entry and put the new one there (replace old offset by new offset, done). This does however mean the old entry will still be in the file until you "garbage collect". That's if the new version of the file is bigger than the old one of course, if it's smaller or the same size you can just overwrite the old entry and not touch anything.

One could even imagine padding files in the archive so there's some leeway in growing the compressed files, it'd likely depend on the format expression.

This is however the sort of thing SQLite already does for you.

You can append just fine, and SQLite doesn't let you replace arbitrary content either. If the new content fits within the space of what it is overwriting then all is fine (as with zip), but if there is insufficient space then the new data is written at the end (as with zip). SQLite has a vacuum command which creates a new database only with used content.

Also done with CBR / CBZ files (comic book archives) -- basically a renamed .zip of images with filenames numbered in ascending order.

The file extension actually makes a difference here, CBRs are RAR compressed archives.

Hence "basically" =P

Exactly. An interchange format should be editable with standard tools included with windows, and it should be easy to write all the code needed to extract part of the file yourself. Sqlite has many benefits, but realistically the only way to interact with it is through someone else's code. That rules it out as a suitable interchange format.

Any performance benefits sqlite might have are easily outweighed by the ease of interacting with the zip-based format.

Reply on news.ycombinator.com