A parallel implementation of gzip for modern multi-processor multi-core machines
github.com
github.com
--rsyncable support too.
On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/
With both pbzip2 and lbzip2 I got errors every now and then while everything worked with bzip2. YMMV.
Also note that bzip2 is block based (due to bwt) and thus does not compromise compression ratios like parallelized gzip implantations.
At 32 cores you can saturate Gbit links (even with good compression ratios!). I hope we will see some more bzip2 love in the future due to it's perfect fit for parallelism.
But those have been a few years ago, so my best advice would be: retest.
You are testing your archives anyway, right? ;-)
Umm no, I handle errors which I expect to be reported to me.
Many decoders simply stop reading after decompressing the first 900k without so much as a peep that something that may astonish you has happened.
[0] http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html#hl-interface
[1] https://docs.python.org/2/library/bz2.html#bz2.BZ2Filehttps://github.com/mgerdts/pigz
This feature is present in Solaris 11.3 and later. Actually, it is in some later patches to 11.2 as well. I added it to speed up suspend and resume of kernel zones.
If you are looking for details of the design of pigz, there is a very well-documented overview in the source of pigz.c:
https://www.amazon.com/Interfaces-Implementations-Techniques...
# time pigz -c /tmp/a.txt > /dev/null
real 0m19.352s user 1m16.148s sys 0m0.344s
# time gzip -c /tmp/a.txt > /dev/null
real 0m47.093s user 0m46.940s sys 0m0.104s
Second, and this isn't meant to be a critique (I'm just trying to understand phenomena I see), is there a reason you prefer presenting it as a percentage decrease? Every time I read "X% decrease" I feel obliged to read the source numbers because I'm never sure if the person is using the terminology correctly or not (you are), since so often people mess that up. For myself, I generally use "X ran in Y% of the time Z took." specifically because I don't want people to misinterpret. Is the "X% decrease" presentation preferred/taught, or considered standard? Am I alone in feeling it's more likely to be misinterpreted?
(Sorry your comment is the one I brought this up on, I've just been wondering this for a while.)
My preferred way of communicating this concept is "we observed a +10% change in X compared to Y." I always use a +/- sign and this helps signal that I'm talking about a relative change.
If I am comparing percents, I'll always specify "relative" or "absolute" change though I prefer to use relative change. Occasionally if the change is small, I will use basis points instead of percents to communicate absolute change.
I think teej explained it well. If I say "the new value is +20% or -20%", it is immediately obvious those have the same magnitude and opposite direction. But, for some people, when I say "the new value is 120% or 80% of the old value", it is not immediately obvious that they have the same magnitude. It requires a small extra step for the reader to realize that this means the same amount of relative change.
The input blocks, while compressed independently, have the last 32K of the previous block loaded as a preset dictionary to preserve the compression effectiveness of deflating in a single thread.
That above is not what the man-page says. The block size for any given session is fixed, so you know the boundaries of each block prior to compression.
Each block has a copy of the last 32kbytes of data at the end of the prior block.
The algorithm used by gzip compresses by finding repetitive strings in the last seen 32kbyte window of uncompressed data, so there is no compression dependency between blocks, even with a copy of the last 32k of uncompressed data from a prior block being present for the current block.
So as long as each parallel block has pre-pended the final 32k of the prior block, the output from the compression algorithm will be identical between a straight sequential compression and a pigz parallel compression. Because at byte 0 of the current block, the 32k window of the uncompressed prior block is available to perform matching against, just as if it were running sequentially.
The only growth from pigz comes from needing to round each parallel compressed block up to a multiple of 8 bits (the huffman codes that are output are bitstrings that don't match with 8-bit byte boundaries). But worst case that is 7 bits per block for each parallel block. Given the performance gains on multiple CPU systems, a few hundred bytes net increase is not likely to matter. If those few hundred bytes did matter, then one should use bzip2 or lzip or xz and get much higher compression ratios (at the expense of much longer time).
(it's relatively easy to remember those commands)
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731634
Another nice thing about pixz is it does parallel decompression, as well as compression.
(Disclaimer: I'm the original author of pixz.)
With large files, this approach would be of huge value. If the files tend to be no larger than block_size - 512, there will be no speedup.
Of course, this would need to be implemented directly in tar, not by piping the output of a decompression command through tar.
The workload I was interested in (compressing virtual machine memory during suspend) tends to be I/O bound. pigz struck a good balance to make it so that cpus and disks both stayed busy in an important subset of the machines of interest. This balanced saturation seemed to be optimal to minimize the suspend time. If the suspend image was on fast storage (that is, could saturate a 10 gig link) multi-threading uncompression for resume made a big difference.
I tried pixz and I think pbzip2 and found that while the compress ratio was better than pigz offers, the delta in CPU-bound run time during compression was unacceptable.
The scales may tip in favor of pixz when compression speed is less important than (e.g.) minimizing the amount of bandwidth required for 1000's of downloads of the compressed content.
split -l1000000 --filter='gzip > $FILE.gz' <(zcat large_fille.txt.gz)
Text log file and database dump in text usually generate very large text file, lzip can reduce their size by a factor of 5 to 10, compared to gzips.
`parallel gzip ::: file1 file2`
I wish there was a standard for this type of stuff. So that an app will check for existing child spawns and act accordingly (IPC).
parallel --pipepart -a bigfile -k gzip > out.gz
The last is faster.
> time gzip -v xubuntu-16.04-desktop-amd64.iso
xubuntu-16.04-desktop-amd64.iso: 1.5% -- replaced with xubuntu-16.04-desktop-amd64.iso.gz
gzip -v xubuntu-16.04-desktop-amd64.iso 119.07s user 4.66s system 98% cpu 2:05.22 total
> time pigz -v xubuntu-16.04-desktop-amd64.iso
xubuntu-16.04-desktop-amd64.iso to xubuntu-16.04-desktop-amd64.iso.gz
pigz -v xubuntu-16.04-desktop-amd64.iso 128.19s user 6.64s system 184% cpu 1:12.97 total