That sounds fantastic. What is the porting process generally like? Are there any possibilities to create an API compatible wrapper to make it a drop in replacement for zlib?
* If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm.
* Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm.
* The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support.
* Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1]
[0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper
On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?
* The larger services require tuning to get the best performance out of zstd, and we use some advanced options.
* We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper.
* We have our own automatic decompression framework that handles many algorithms [0].
* A lot of use cases switched over from other algorithms than zlib.
* A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win.
[0] https://github.com/facebook/folly/blob/master/folly/compress...
https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6...
But all this depends on where the bottleneck is in any particular case. My VMs are on HDD, so increased decompression speed doesn't matter that much, but reduced size helps reading the necessary data faster. Linux VMs seem to be more compressible than Windows ones.
It's certainly better than zlib in any case.