Hacker News Reader

Comment by _wmd | Hacker News Reader

Zstandard is awesome, there is nothing else close to it that I've seen. There are plenty of codecs that give you either obscene ratios or low CPU usage, but none that support both in combination. Zstandard on its highest setting easily competes with xzip for anything I've thrown at it. Decompression throughput is unaffected by compression setting -- it just burns more memory. So you can get xzip-like ratios with decompression throughput approaching 1GB/sec

It allows trading lower CPU for gzip-or-worse compression, and you can mix the settings within a single file. This means you can e.g. use the lowest setting (or no compression at all - it supports that too) to append to a file, while occasionally recompressing recent appends into a single block using the highest setting -- so the cost of compression can be amortized

The only petty annoyance with it is ecosystem support - e.g. GNU tar has no option for it, so it's slightly more painful to work with

View on HN

tar-1.3.1 added support for zstd with the option `--zstd` and `-a`, the auto-decompression flag, also supports zstd. Older tar versions also have the `-I` flag which you can use to (de)compress with zstd.

We're working to improve zstd support in the ecosystem over time, but this work moves slowly, and it takes a long time for upstream work to make it to users systems, especially LTS systems.

> tar-1.3.1 added support for zstd with the option `--zstd` [...]

Is there a way to pass options to zstd when used like this?

Last time I checked zstd does not support getting options from an environment variable (unlike .e.g. gzip).

With a new enough version of tar you can pass a command with arguments to '-I' (e.g. `tar -I 'zstd -19' ...`).

The alternative is piping the output from tar through zstd yourself.

I build my own custom tar 1.32 rpm with zstd support https://community.centminmod.com/threads/custom-tar-archiver... and can confirm you can pass options that way.

Starting with zstd-1.3.8 we support the `ZSTD_CLEVEL` environment variable. We've started with a small scope for the variable, because we don't want users to unexpectedly remove the source fie, for instance.

If you want to pass extra options, you can pipe the output to zstd, which is exactly what tar is doing internally.

> nothing else close to it that I've seen

My understanding is that Oodle generally performs better (less CPU on both ends for any desired compression ratio) than ZStd in pretty much every context,

http://www.radgametools.com/oodle.htm

... assuming you control both compression and decompression and are willing to spend money for it.

I don't see comparisons against any version of Zstandard, much less Zstandard 1.4, there.

(Besides, regardless of the state of benchmarks today, the momentum is clearly with Zstandard, with Facebook, Intel, and the open source community behind it. The basic lossless compression algorithms haven't changed much since the late 1970s. Making compression fast is mostly just a long slog of engineering hurdles, the kind that big companies are very good at doing.)

http://cbloomrants.blogspot.com/2018/06/zstd-is-faster-than-... is versus zstd 1.3.3. There are others too on that blog.

Oodle and zstd are both very impressive, it's a shame the former is not free.

Zstandard uses ANS in some (all?) of its modes, that's very much brand new -- 2014

> regardless of the state of benchmarks today, the momentum is clearly with Zstandard, with Facebook, Intel, and the open source community behind it.

This argument by hand-wave doesn’t match up with demonstrated progress over the past few years.

Irrespective of the skill or insight of its individual engineers, I would be surprised if a schizophrenic hack-it-with-duct-tape kind of engineering culture like Facebook could keep up with a focused and motivated expert like cbloom over the medium term (though I suppose the latter could conceivably at some point lose interest in the domain and switch to building something else).

I've seen this play out before. The history of jemalloc, particularly how it rapidly outpaced virtually every other allocator while under development at Facebook, suggests otherwise. At this point jemalloc is so good that there is little reason other than NIH to use anything else (unless you want an especially hardened allocator for security and are willing to give up some performance). Even Google uses it in Android.

Zstandard is likewise deservedly on track to dominate the lossless compression space.

> At this point jemalloc is so good

Some folks disagree:

> [Ruby's] memory usage only reduces when using jemalloc 3; memory usage is still high when using jemalloc 5. Nobody knows why, so that makes the choice of defaulting to jemalloc very dodgy.

via https://www.joyfulbikeshedding.com/blog/2019-03-29-the-statu...

Zstd is from Facebook, sure, but to be more specific its development is lead by Yann Collet (of LZ4 fame) who is unquestionably a focused and motivated expert.

But it’s not WinZip or WinRAR. This is a product marketed for professional use(r)s. How could it ever become more popular than something you just have?

Could be that Zstandard outperformes it but I also have similar good experience with brotli.

Good compression speed ratio (use q 1 for brotli lower than 0.6):

    tar -I"brotli -q 2" -cvf file.tar.br inputfiles

Decompression:

    tar -Ibrotli -xvf file.tar.br

It outperformes gzip and is near xz region without a lot of CPU power and is also blazing fast. Really useful if you have e.g. a big PostgreSQL db dump which you want to transfer to your own machine. Examples for Postgres dumping and restoring:

    pg_dump adatabase | brotli -q 2 > dump.sql.br
    brotli -d < dump.sql.br | psql -U postgres dbusernamehere

I got to experiment with it on some pretty poor spec MIPS processors not long after it was made public. Even there, on architecture it wasn't designed of specifically optimised for, it seriously outperformed competitors.

In my experience, for the best balance between compression speed and compression ratio, nothing beats 7zip with the right options:

  -mmt=$(nproc) # use all available cores
  -ms=off       # disable solid archives (compress each file separately)
  -m0=lzma2     # lzma2 has better threading than lzma1
  -md=64m       # dictionary size
  -ma=0         # "fast" mode
  -mmf=hc4      # hash chain match finder
  -mfb=64       # number of "fast bits"
  -mf=off       # disable filters

The biggest gains are: 1) using all available cores, 2) setting the match finder (the binary tree match finders are terribly slow; I haven't played much with the newer patricia tree match finders), 3) disabling solid archives (this seems to cause 7zip to distribute the work more evenly between cores, though it still may only use a few cores if there are many small files), 4) using "fast" mode (whatever that is, it gives a noticeable performance boost and doesn't seem to affect compression ratio much).

Every few years I try zstd and others, and for the data I work with (primarily a mix of json and fixed-width-field binary data), lots of tools beat 7zip out of the box, but they fall short of 7zip with the above command-line options.

Perhaps it's the use of a dictionary? As far as I'm aware, tar, zstd, xz do not use one by default, it's an extra set of hoops to create a training set, create the dictionary, use it for compression, and pack it away somewhere so that it's available for decompression, and then actually use it for decompression. If that's all being done by 7zip just by passing -md=64m that's pretty cool.

Edit: Ahh, I was confused. Neither require a separate training step. Zstd offers an option to do a training step. Both always use dictionaries with a default size that can optionally be changed.

A comparable zstd call that uses a 64 MB window size and all cores is:

    zstd --long=26 -T0

From there you can tune the compression level, or increase the window size up to 2 GB (--long=31). zstd won't beat the compression of xz, but it can compress much faster if you trade off some space.

Well, it didn't work that well for me. Adding to https://news.ycombinator.com/item?id=19682019

    $ time 7zr a -mmt=$(nproc) -ms=off -m0=lzma2 -md=64m -ma=0 -mmf=hc4 -mfb=64 -mf=off linux-5.0.8.tar{.7z,}
    real 60.49 user 158.94 sys 3.06 maxrss 8995040

    $ stat -c '%s %n' linux-5.0.8.tar.7z 
    127700475 linux-5.0.8.tar.7z

    $ time 7zr e -so linux-5.0.8.tar.7z >/dev/null
    real 14.09 user 13.96 sys 0.12 maxrss 282208

Basically:

- it took twice the time to compress data even compared to xz -2 (which also uses lzma2 under the hood),

- it is comparable to zstd/bzip2 ratio-wise,

- it used almost 6 times (!) more RAM than even zstd -12 --long,

- it only used about 2.5 CPU cores out of 4 while compressing (which aligns pretty well with your reasoning for using -ms=off).

----

But hey, source code is not that regular. Since you mentioned JSON and fixed-width-field binary data, I decided to re-run benchmarks on 10M lines of nginx access logs: they're way more regular in their structure (repetitive URLs, timestamps, Mozilla/5.0, stuff like that) that might benefit from larger window sizes.

    $ time lbzip2 -k access-log-10m.log 
    real 90.59 user 313.04 sys 18.46 maxrss 117904

    $ time ~/zstd-1.4.0/zstd -T0 -k -12 access-log-10m.log -o access-log-10m.log.zst-12
    real 77.34 user 277.21 sys 1.55 maxrss 886416

    $ time ~/zstd-1.4.0/zstd -T0 -k -12 --long access-log-10m.log -o access-log-10m.log.zst-12-long
    real 69.24 user 242.18 sys 1.85 maxrss 1411872

    $ time 7zr a -mmt=$(nproc) -ms=off -m0=lzma2 -md=64m -ma=0 -mmf=hc4 -mfb=64 -mf=off access-log-10m.log{.7z,}
    real 109.10 user 356.42 sys 4.69 maxrss 9777664

    $ stat -c '%s %n' access-log-10m.log* | sort -n
    208537395 access-log-10m.log.bz2
    231953002 access-log-10m.log.zst-12-long
    237566691 access-log-10m.log.zst-12
    249412192 access-log-10m.log.7z
    3386733539 access-log-10m.log

Now tweaked 7z did better CPU- and time-wise, but it's still behind zst and bz2 on every metric, especially RAM which it requires so much of (literally gigabytes) it becomes impractical in a number of situations. And we needed a pretty regular input (not just some pretty compressible text like source code or Wikipedia dump) to close that gap. So I can't really recommend your suggestion, unless you have some niche input that benefits from that particular set of options (but then, who has time to learn lzma internals and how every option plays with different kinds of input?).

----

It's also worth pointing out that zstd also has plenty of options to fiddle with: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...

> There are plenty of codecs that give you either obscene ratios or low CPU usage, but none that support both in combination.

I still find lbzip2 (which is a bzip2 reimplementation with better algorithms and support for multithreading) quite competitive for highly compressible data. Here's quick and unscientific test that shows that lbzip2 (-9) is still both faster and has better ratio than zstd (-12, --long or not) while also using the least amount of RAM (tmpfs, multithreaded compression using 4-core Xeon E5-2603 v1):

    $ time lbzip2 -k linux-5.0.8.tar 
    real 24.21 user 84.87 sys 4.07 maxrss 105472

    $ time ~/zstd-1.4.0/zstd -T0 -k -12 linux-5.0.8.tar -o linux-5.0.8.tar.zst-12
    real 30.69 user 105.27 sys 0.61 maxrss 942416

    $ time ~/zstd-1.4.0/zstd -T0 -k -12 --long linux-5.0.8.tar -o linux-5.0.8.tar.zst-12-long
    real 31.28 user 107.90 sys 0.86 maxrss 1532432

    $ time xz -T0 -k -2 linux-5.0.8.tar
    real 34.40 user 123.59 sys 0.57 maxrss 410192

    $ stat -c '%s %n' linux-5.0.8.tar* | sort -n
    126382954 linux-5.0.8.tar.bz2
    126394210 linux-5.0.8.tar.zst-12-long
    128003669 linux-5.0.8.tar.zst-12
    131418488 linux-5.0.8.tar.xz
    863426560 linux-5.0.8.tar

The only clear advantage zstd has is decompression speed:

    $ time xzcat -T0 linux-5.0.8.tar.xz >/dev/null
    real 17.25 user 17.06 sys 0.17 maxrss 17312

    $ time ~/zstd-1.4.0/zstd -dc -T0 linux-5.0.8.tar.zst-12 >/dev/null
    real 2.08 user 1.97 sys 0.08 maxrss 27088

    $ time ~/zstd-1.4.0/zstd -dc -T0 linux-5.0.8.tar.zst-12-long >/dev/null
    real 2.26 user 2.03 sys 0.17 maxrss 535360

    $ time lbzcat linux-5.0.8.tar.bz2 >/dev/null
    real 10.34 user 33.74 sys 3.53 maxrss 127088

Given that it's the decompression speed that is typically the "user-facing" part in many contexts (with compression being done by automatic jobs etc), such a difference in decompression speed is pretty awesome indeed.

(which is also what makes it a near-perfect codec for HDF5, via blosc-hdf5)

I mostly use lbzip2 for day-to-day tasks too. But shouldn't you compare to pzstd to be fair? It's not very surprising that running 4 threads is faster than 1 thread in wall clock time.

EDIT: no, I was wrong, `zstd -T0` is basically the same as `pzstd`.

I'll just add that I tried it and pzstd -12 is still significantly slower than lbzip2 -9 on my machine, with approximately the same compression ratio for linux-5.0.8.tar.

EDIT: no surprise, as -T0 also enables multithreading.

Reply on news.ycombinator.com