Hacker News Reader

Comment by charlieyu1 | Hacker News Reader

Not a fan of zipping everything.

One thing I missed about working with old .doc files instead of .docx, was that it was very fast to search a folder with hundreds or thousands of files for a specific word. Not possible for zip formats. (I just saved a file in .doc right now and open it with a hex editor and it does contain the contents in plain-text.) It is a problem to search zip formats or pdf files with grep, and I really doubt that the zipping is necessary for text with maybe a few thousand characters.

View on HN

ZIP is a real simple format under the hood.

There are zip-aware grep tools that you could investigate if they fit your workflows.

There's also nothing stopping you from storing the contents of zips as unzipped directories and rezip when/as necessary. (I wrote a tool called musdex years ago to automate that flow in the context of source control: source control the contents of something like .docx expanded into a full directory structure, but preserve the ability to "double click the Word file to edit".)

As a wrapper of "collections of deeply related files, some which may be text and some may be binary", ZIP is one of the better choices that we have (compare to TAR or MIME envelopes, for instance).

Zip files don't need to be compressed, if there's a need maybe you can promote adding the option upstream to whatever apps you use (if it's not already there somewhere). Or a .zip equivalent for zgrep.

Though pattern matching in files feels very fragile - simple text patterns would still rely on the text being kept in a contiguous chunk and no embedded data/markup within the section you're searching for...

Though I generally see the prevalence of zip files a result of people assuming that collections of files/streams need to be packed into a single object for "user simplicity", but we already have this with OS support in the form of directories. It just seems people have embedded the assumption that tools and UI handle directories as single units poorly, when that doesn't really need to be the case.

Directories still have problems when it comes to user simplicity. For example, try emailing one.

Fundamentally, directories are problematic because they are user-facing. When used as a file format, you risk running into situations where the user has (sometimes inadvertently) deleted some of the files or edited them, partial copies etc. Making it all into a single zip file makes it clear that it really should be treated as a single atomic unit by any casual user, while still allowing power users to hack on it when needed.

Browsers don't really have good support for downloading a whole directory.

Have you ever tried to download 100 individual files or moved 10,000 files to a drive VS a singular zip? The difference is night and day

Just use zipgrep to search inside zip files? Or just unzip it then use regular grep. I don't think it makes sense to gimp a file format just to make life slightly easier for people using archaic Unix utilities.

.zip files specify the compressor used for each file, one of which is no compression. You have to explicitly tell it to produce such a suboptimal result but it can be done.

Going back to the 90s--I was using .zip files as a container to keep all files in an order together. As time went on it the requirements included updating some of those files *in a shared access environment*. The workflow didn't permit two people to be updating the *same* file, but simply rebuilding the file on update was not permissible for concurrency reasons (nor was it permissible for speed reasons--I was state-saving each item checked off on a screen.) Solution: The files which could be updated in use were forced to no compression. I then wrote my own code which read the header and got an offset within the .zip that was the body of the file. I then treated it as a simple record-based file with that offset and wrote my changes. The CRCs were recalculated and written if the user left the screen, if the user crashed out (this was the DOS era and the machines were on a factory floor) the file would be reported as corrupt but would actually be intact. My program didn't check the CRC and would perfectly happily handle the situation. Ideal? No. The best that was realistically possible at the time, I believe so.

JAR and APK files are really ZIP, as is EPUB.

True, ZIPs are hard to grep. However, gzip and progeny come bundled with convenience scripts as zgrep, bzgrep, xzgrep, etc.

Maybe use a grep-friendly format if that's important to you, or otherwise a filesystem with transparent compression (btrfs, ZFS, or [gasp] NTFS).

To be fair. Iirc, an epub is stored uncompressed. So you should be able to get a directory of them no problem.

Reply on news.ycombinator.com