terrelln

The x axis is compression speed, and the y axis is compression ratio.

Zstandard outperforms zlib in compression ratio, compression speed, and decompression speed (not shown). The only reason to stick with zlib is for compatibility with systems that expect zlib.

GPLv2 was added before the patents grant was removed, so zstd could be included in the linux kernel.

If you're using lrzip, you should also check out zstd long range mode [0]. It uses a long window (128 MB by default, up to 2 GB), together with an efficient search strategy, and multithreading. For example, a 2 GB window, with 4 threads, at level 10:

    zstd --long=31 -T4 -10

It should be faster than lrzip + zstd and provide about the same results.

[0] https://github.com/facebook/zstd/releases/tag/v1.3.2

Btrfs already supports multithreaded compression and decompression. Each 128 KB block is (de)compressed with a single thread, but multiple blocks can be (de)compressed in parallel.

The zstd CLI supports multithreaded compression with the flag `-T <num-threads>`.

I implemented zstd compression in btrfs/squashfs, and work on upstream zstd.

* Btrfs compression is multithreaded, and can use up to the number of cores available on the system. * Compression might not help speed on SSDs, but it should help reduce the burn rate, if you care about that. * My intern wrote a patch to add compression level support to zstd in btrfs [1]. It should be merged upstream soon. * Slightly off topic, but grub will soon understand btrfs compressed with zstd.

[1] https://lore.kernel.org/linux-btrfs/20181031181108.289340-1-...

Both PEXs and XARs package a python script and its dependencies in single hermetic file.

PEX is a self-extracting zip file which has to be fully extracted before being run. The extracted files could potentially be modified.

XAR is a self-mounting compressed SquashFS filesystem image. SquashFS will decompress pages lazily and cache the result in the page cache, so the startup time is much faster. Since SquashFS is read-only, the files can't be modified.

I spent some time today investigating what exactly is causing the difference between native and XAR start times. I confirmed the culprit is `pkg_resources.load_entry_point()`. Modern installations using wheels should avoid this overhead, and those native installations will be slightly faster than XARs:

black: 0.171 s (vs 0.208 for XAR) jupyter: 0.165 s (vs 0.179 s for XAR)

My test setup used the older loading method because "pip install ." won't install wheels if the wheel package isn't installed in the virtualenv.

Facebook's PAR is a self-extracting zip file, I assume Google's is similar. XARs are self-mounting SquashFS archives (a compressed read only filesystem). This means that XARs don't have to be extracted to a temporary directory to run, they can run in place. Zip files have to be completely extracted before running, but SquashFS decompresses pages on the fly, so startup times are much faster (especially with zstd compression).

We currently don't have a nice open source API for building node apps, but would welcome PRs that get us in this direction!

There are two ways to build a node app using the XAR builder tools. 1. Use the `make_xar` tool which will create a XAR from a directory and takes an optional script to run on execution. 2. Use the XAR builder library to make a XAR builder that is specialized for building node apps.

The current Python XARs rely on Python being on the system path. But it would be easy to build a custom Python XAR with the XAR builder library that includes the Python executable and makes sure to use the packaged executable.

Admittedly I haven’t profiled this yet, but my guess is it is a constant overhead of setting up pkg_resources that the native code uses to load the entry point.

The test against native start speed was hot, so the pages required were already in the page cache, so the filesystem shouldn’t matter.

XARs are just self mounting compressed readonly filesystems with an executable inside. We get hermitic dependencies by setting the PYTHONPATH, LD_LIBRARY_PATH and such in the bootstrapping script.

One big benefit is that the filesystem only decompresses the pages as needed, which greatly improves the start up time over existing solutions.

Zstandard version 0.5 (and all other versions < 0.8.1) were pre-releases where forward compatibility was never intended. However, Zstandard is still backward compatible with versions down to version 0.4. All releases since August 2016 have been forward and backward compatible.

It is unfortunate that Xenial picked up version 0.5, but Yann Collet worked hard to get version 1.3.1 backported for this exact reason https://bugs.launchpad.net/ubuntu/+source/libzstd/+bug/17170....

It is mentioned at the bottom of the README, but it isn't very findable. I've opened a PR https://github.com/facebook/zstd/pull/1085.

The GPLv2 license was added for inclusion in the kernel, you may choose either GPLv2 or BSD. See the header file https://github.com/facebook/zstd/blob/dev/lib/zstd.h.

The library is dual licensed under plain BSD [1] and GPLv2 [2].

[1] https://github.com/facebook/zstd/blob/dev/LICENSE [2] https://github.com/facebook/zstd/blob/dev/COPYING

Zstandard also maintain ABI stability for a portion of the API, and require a macro definition to access the unstable parts.

Newer versions are forwards and backwards compatible with older versions. The compression format stabilized starting with version 0.8.1.

Generally, pages in long ranges will be accessed less than those in a short ranges, and pages beyond the window size will never be accessed. You can construct data that doesn't fit this pattern, but its probably a good bet to make. With a 1-2 GB window size, I would expect there to be large chunks of the window that are rarely accessed.

The long range matcher has a configurable match size, where it will only look for matches that are at least that large. By default it is 64 B, but by making it larger, say 4 KB, you can ensure that if you are going to force a page fault, you get enough benefit.

Upstream SquashFUSE also has zstd support.

https://github.com/vasi/squashfuse

We intend to improve the interaction of multithreaded compression and long range mode in a future release, as they go hand in hand. We also intend to support mmap in the CLI, which should help reduce the memory usage for very large window sizes when writing to a file.

It was relicensed in v1.3.1. It is now dual licensed under BSD without the patents clause [1] and GPLv2 [2]. GPLv2 was added for inclusion in the Linux kernel.

[1] https://github.com/facebook/zstd/blob/dev/LICENSE

[2] https://github.com/facebook/zstd/blob/dev/COPYING

Zstd recently added a long range mode that can find matches up to 2 GB in the past using a specialized algorithm [1]. It can be enabled on the command line with `zstd --long` for a 128 MB window, or `--long=windowLog` for a `2^windowLog` byte window.

[1] https://github.com/facebook/zstd/releases/tag/v1.3.2

lz4 and zstd are complementary, not competitive. lz4 targets applications where extremely fast decompression speed (and compression speed) are required, at the expense of compression ratio. However, many applications would prefer to spend a bit (or a lot) more CPU in exchange for better compression, while maintaining fast decompression speed.

lrzip is a preprocessor that finds matches in the distant past that the backend compressor (xz) couldn't normally find. zstd has a new long range matcher mode inspired by the ideas behind rzip/lrzip with some extra tricks. It produces data in the standard zstd format, so can be decompressed the the normal zstd decompressor. There is a short article about it in the latest release notes https://github.com/facebook/zstd/releases/tag/v1.3.2

zstd uses ANS [1][2] to encode its (literal length, match length, offset) triples.

[1] https://github.com/Cyan4973/FiniteStateEntropy

[2] https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi...

How are you compressing the data?

I would expect a dictionary to be useful if the data is broken into chunks, and each chunk is compressed individually.

If the data is compressed as one frame, I would be very interested in an example where the dictionary helps.

What use case do you have in mind for packaging dictionaries with archives? There is an ongoing discussion about a jump table format that could contain dictionary locations [1].

[1] https://github.com/facebook/zstd/issues/395