Hacker News Reader

Comment by UI_at_80x24 | Hacker News Reader

I have been incredibly impressed with Zstandard, and if you are not currently using I suggest giving it a try.

I'm a sysadmin so this has worked it's way into every corner of my servers, from backups, to log compression, and more. Most of my servers run FreeBSD, and with the merge of linux/FreeBSD ZFS I am looking forward to the integration of ZSTD into ZFS so I can use that for on-the-fly Zpool compression.

I can safely say that ZSTD has saved me TB of space over LZ4 & GZIP.

View on HN

I used to be a big fan of XZ(which uses lzma2), but Zstandard has good-enough compression at high levels with orders of magnitude faster decompression. It's also more versatile in its compression levels. It's now also my go-to for new projects.

XZ/LZMA2 has the problem that it slows down a lot on incompressible data and can even choke pretty badly on compressible data, which Yann Collet's algorithms (LZ4 and ze standard) don't do nearly as much. This means zstd can be used indiscriminately, while LZMA would frequently need some sort of qualifier to avoid processing incompressible data and the associated loss of performance.

To put some numbers to this, on the machine (>10yo laptop) I'm typing this on:

              trivial    random     .wav file
     LZMA     ~20 MB/s   ~2 MB/s    ~1 MB/s
     zstd     ~500 MB/s  ~250 MB/s  ~70 MB/s
     zlib     ~40 MB/s   ~13 MB/s   ~6 MB/s

(xz is a compression container format that doesn't imply any particular compression algorithm. It has a concept of "filters" that are applied in "chains" over "blocks". One filter happens to be "lzma2", but to say xz "uses lzma2" is a bit of a confusing mental model, as xz could just as easily "use zstd" if someone allocated a filter ID for such.)

In principle yes, but in practice the only filters included in my version are lzma1, lzma2, a basic delta filter and a few filters for executable code. As far as I can tell all the presets that are reachable without explicitly specifying a filter chain are just lzma2 with various options set.

I would love if xz had a zstd filter and a filter that adds/processes some error correction code. But with what's there right now I think it's fair to treat it as a pure lzma2 container format.

For system backups and logs and stuff that rarely need to be decompressed, but stored for a while and then deleted in rotation, I still strongly prefer xzip. It's much slower to create a .tar.xz file with the default settings, but also saves a considerable amount of disk space.

Side by side comparison right now between a full system file tree backup of a mediawiki server, the .tar.xz file is about 1.8GB, the .tar.zstd file is 2.15GB. The same thing in traditional .tar.gz format is obviously even larger.

From the research I've done on common uses of zstd it's much better suited to applications where very high speed of compression and decompression is a major requirement, but absolute smallest file size is not. Typically as a replacement for gzip.

Try xz -1. Faster than gzip, smaller size. We use it for regular backups. Although the xz format is not recommended for long term archiving because it's unsafe.

https://www.nongnu.org/lzip/xz_inadequate.html

I'm going to read this in more detail later today when I have time. But I just skimmed it, and most of the points about xz being fragile and having a weird format would seem to apply in a scenario in which a .tar.xz file became corrupted or bit-flipped from its original? eg: in which the .xz file, after some long period of archiving, no longer matched the same sha256sum as when it was originally created. Thereby making it difficult or a huge hassle to get authentic lossless/uncorrupted data out of it?

The "unsafe" thing is nonsense. The expressed concern is how the contents degrade in the presence of arbitrary bit flips. Any highly-compressed format is going to be susceptible to corruption amplification from random bitflips, and the higher compression level, the larger fallout. Ensure your compressed archives are not corrupted using some external FEC; either in your hardware, or something like par2.

Those are with the default settings but can't you increase the compression ratio target of zstd to match xz and still be significantly faster?

In my tests with the mix of files on this particular virtual machine, increasing the compression ratio of zstd to achieve an equally small output file size as xzip, it's equally as slow on a per-core basis.

It does have a performance advantage in that it by default uses both CPU cores, while a default tar with -J option will use xzip on only one CPU core.

If the compression ratio is about the same, I'd choose zstd. The decompression performance is much better than xz.

And zstd keeps evolving contrary to xz, which means they'll outperforms xz compression ratio in the following years

We can hope!

xz on highest (normal) compression level easily beats the compression ratio of zstd on highest compression level (--ultra -22) on any data I've tested. However with xz reading the compressed files easily becomes a bottleneck, zstd has great read speeds regardless of compression ratio.

In my experience (a couple years ago, YMMV) high-level (22) zstd compression is about as slow as high-level xz (9) compression and the results are slightly larger. High-level zstd decompresses vastly faster than xz — so I would use it for things like CDNs, where users are decompressing contents much more frequently than compression — but for backups, which are decompressed zero or one times, it probably doesn't matter.

This is basically what arch linux has done with their repo/packages.

> zstd and xz trade blows in their compression ratio. Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup.

In some things it's often a trade off of expected network speed vs cpu power. On very limited network speed connections such as a last mile ISP connection in a developing nation, where speeds might be under 5Mbps downstream, xz has some advantages in lower transfer times.

Often times persons on very limited bandwidth connections (even a Hughesnet or Viasat consumer grade satellite link in a very rural area of the USA) will have more than sufficient CPU to decompress xz in a reasonable amount of time. Such as a five year old core i5 desktop computer.

In environments where it can be expected that the bulk of users are on very fast broadband connections, zstd and slightly larger file sizes have an advantage.

This (purported) benefit is bounded by the difference between compression ratios. You're responding to a claim that the difference was 0.8%. This is pretty marginal — that first 99.2% takes exactly the same with xz or zstd, and the remaining 0.8% is not large in comparison.

Take for example ~13.5 GB. This is a 6 hour download at your 5Mbps figure. An extra 0.8% is less than 3 minutes on that 6 hour download.

For the corpus of things I've dealt with recently, the difference is a lot more than 0.8% in size. As much as the difference between an entire compressed virtual machine (copy of a set of LVM logical volumes that have been copied with dd and compressed) being 4.2GB vs 5.15GB in size.

If it might take ten hours to transfer 900MB in some unusual environments, that's a big savings.

Also there is a degree of asymmetry, in that it's not so difficult for the system creating the xzip archive to have a lot of CPU (four CPU cores out of 12 or 16 on a dual socket current generation xeon system, or a threadripper). Since xzip is much more time consuming to create than to extract.

Or for any other environment where WAN connectivity is extremely limited in throughput.

I'd like to see some real example where the compression difference between xz and zstd is actually 4.2 vs 5.15 GB. That's huge and bigger than any difference I've ever seen published between the two. Are you sure you actually used comparable settings with zstd? IMO, this is not a typical difference and shouldn't guide intuitions about where to use one or the other.

I wonder if software compression can now cost compete with compression cards like Intel QuickAssist on constantly loaded servers.

Compressing large files/backups using all available cores at the highest setting is a bliss. Sadly the choke point now is encryption. Hopefully some kind soul will some day add this to gpg or equivalent. There are parallelizable cipher modes like CTR and GCM.

https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation

If your need of encryption is related to backups, look into "Borg backup". It can handle both encryption and zstd, also do dedupe natively since it's storing data as blocks.

Borg only utilizes one thread and does not support multithreaded zstd, so the write throughput will likely be (much) lower than ZFS -- typically 60-120 MB/s. Since ZFS can also dedupe, Borg might not even be faster for heavily deduped backups.

ZFS dedupe is broken and I don’t recommend using it unless you really know what you’re getting into.

Broken in what way?

It uses a ludicrous amount of RAM and a fixed block size and offset boundry, making it very slow and fairly ineffective.

I don't think zstd makes sense for transparent disc encryption (compared with lz4). The table at [1] shows that zstd is half the speed of lz4, and provides only a modest improvement in compression ratio.

1. https://facebook.github.io/zstd/

What kind of compression level do you use?

You probably have some conditioning that discourages use of '-9', but that's generally totally fine to use with zstd in terms of speed.

I've been a zstd convert for a few years now and haven't settled on any particular ideal setting. Sometimes the very high compression levels (-19) produce a worse result, sometimes they kill it. I think most of the time I'm using -9.

I've seen cases where '--ultra --long -22' was bringing size down by 10% or more compared to the normal settings, and also cases where changing the defaults at all produced significantly lower compression.

For approximate reference (N.B., this graph is somewhat dated — zstd 1.3.4 is from March 2018): https://raw.githubusercontent.com/facebook/zstd/v1.3.4/doc/i...

(On the dataset used for the figure, zstd -9 was about the same MB/s as gzip -4, and compressed at ~3.5x to gzip's ~3.0x.)

Reply on news.ycombinator.com