Hacker News Reader

Comment by bradleyjg | Hacker News Reader

Gzip files can just be straight concatenated together. There are some caveats around metadata but it’s an amazing property.

View on HN

But only if the first file ends on a byte boundary of course. Just to save someone trying this some debugging :)

> byte boundary

Do GZip files have bit-based lengths?

Yes they do!

Wow, I wonder what led to that design decision. Does that mean you have to take that into account when concatenating them?

Well, the whole article is about the flaws in the format :)

But the answer in this case is Huffman coding. The most common sequences compress to just a few bits, but backreferences and such are not byte-aligned, they just follow literals in the bitstream. Aligning things to bytes would significantly reduce the compression ratio.

Is that the compressed file that ends on the byte boundary, or the uncompressed one?

Compressed. The compressed stream isn't byte aligned. Normally this isn't a problem because an application will flush the output and pad it to end on a byte boundary before closing a file (see https://www.bolet.org/~pornin/deflate-flush.html for a good explanation)

Where this becomes a problem is when someone thinks "I'll compress my logs as I write them out". The next problem is what to do on restart. It's tempting to think "I can just append to the existing file because I can concatenate two gzip files". But if the restart is after a crash rather than a clean exit you get log salad.

Reply on news.ycombinator.com