Parallel Gzip – Pigz | Hacker News Reader

Parallel Gzip – Pigz

leimao.github.io

47 pointskeyboardman22 comments

Open articleView on HN

This is a low-effort post, showing basic CLI usage and nothing else. Pigz does not actually decompress in parallel, as the author implies, underscoring the importance of benchmarking things — especially when you're only considering them for performance reasons. See: https://github.com/madler/pigz/issues/36

For compression, pigz is only useful for compressing a single large file in a scenario where you must use gzip. For large collections of files, wrapping normal gzip with GNU Parallel would be faster. In scenarios where you control both compression and decompression, you'd be better off picking a more modern compression algorithm than gzip. ZStandard is almost always the right choice, but this benchmark is a great resource for comparing other options: https://quixdb.github.io/squash-benchmark/

See also:

pbzip2: https://github.com/ruanhuabin/pbzip2

plzip: https://www.nongnu.org/lzip/plzip.html

And possibly the cleverest of the parallel compressors, tarlz, which, while still being backwards compatible with tar, also allows parallel decompression, something none of the others can do: https://www.nongnu.org/lzip/tarlz.html

Also https://github.com/vasi/pixz, which uses a more computationally efficient yet widely supported compression algorithm (lzma/xz) and adds indexing, making byte range access possible in large files within the archive (and does also support parallel decompression).

Zstandard also has experimental support for seekable (and thus parallel) decompression, but it isn't included in distributed binaries. See: https://github.com/facebook/zstd/issues/395

Note that pbzip2 can only parallel-decompress files originally compressed using pbzip2. lbzip2 (https://lbzip2.org/) can parallel-decompress arbitrary bz2 files, which is a more common need IMHO. bz2 is very CPU- and RAM-intensive, but gives the best ratios of any mainstream algorithm, so it's really only useful for publicly distributed files where bandwidth considerations dominate.

The source code looks a little extreme... Is it auto-generated?

https://github.com/ruanhuabin/pbzip2/blob/master/pbzip2.cpp#...

I doubt it, it does look uglier than modern C but it looks like it's written to work in ANSI-C/C89 entirely without any language extensions since then. That's going to shape things quite a bit (in particular the dense bit of variable declarations at the top of every function) simply because a lot of the things like being able to declare variables later on in the source code just wasn't a thing in that version of the language.

Seems pretty standard to me. (Although I also typically only write C in C89, so modern C might be a lot prettier.)

Where are the 20 line functions, the modularity, the refactored nested if's?

Like I said, pretty standard! ;-)

The bgzip format (widely used in bioinformatics) supports parallel decompression, while being readable by gzip. (Compression ratio isn’t the best, though.)

You basically only need to learn about "zstd -T0". Better and faster algorithm and multi-threading directly included.

sadly also doesn't work for decompression.

Everybody's needs are different, but zstd is so fast the lack of multithreaded decompression doesn't matter very often. I've only ever found parallel decompression useful with bz2 files (which are insanely CPU-intensive to decompress), using lbzip2.

Zstd also has experimental code in the `contrib` dir for parallel (and, more importantly, seekable) decompression: https://github.com/facebook/zstd/issues/395

Thanks for the Information. You're right zstd even on a single core is pretty fast but for multicore systems with nvme drives multithreading would be beneficial.

Looks great! I’m still surprised these tools (gzip, etc) haven’t been update to take advantage of multicore support.

In the gzip case, the reason is that the compressed stream format is inherently sequential: you need to decompress everything before a byte before you can decompress the next byte. So parallizing is not really possible, without changing the format.

Newer compression formats have multiple streams that can be decompressed in parallel, or have blocks that are compressed independently without dependencies (bzip2 can be decompressed in parallel due to independent blocks).

So is Pigz not backward compatible with gzip?

Pigz is backward compatible — it breaks the input into 128K chunks, compresses each in parallel, and concatenates them. Gzip supports naive concatenation of compressed files so the result can be decompressed by standard gzip.

Pigz does not implement parallel decompression, but pigz-compressed files could, in principle, be decompressed in parallel. I assume there's not much urgency for this because gzip decompression is fast enough: >3x faster than compression and >10x faster than bzip2 decompression on default settings. (See: https://quixdb.github.io/squash-benchmark/ )

I don't think Pigz decompresses in parallel.

It's good that they haven't. If you're compressing a large number of individual files (rather than combining them into a single archive), it's still more efficient to run multiple copies of gzip or whatever via something like "parallel".

Good point. I don't think anything bad would happen if you could opt in via something like `-j8`, preserving the default backwards-compatible behaviour and offering an escape hatch for newer workloads / machines.

Reply on news.ycombinator.com