I love everything about zstd. It compresses well, it compresses and decompresses fast.
I wish browsers supported it for content encoding because it's a good deal faster and smaller than gzip, nevermind brotli.
I wish browsers supported it for content encoding because it's a good deal faster and smaller than gzip, nevermind brotli.
I ended up using lz4 instead for my project. Lz4 only added 4kb to my compiled wasm size for an encoder and decoder. And it’s native rust (which makes wasm compilation effortless). Lz4 is a no-brainer to include.
The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.
lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194
There's also LZ4_HC.
It may not offer the same API 1:1, but it has no interoperability issues that I've encountered. I think no one has bothered to implement it in Rust because most use cases don't mind the added bloat you're talking about, not because implementing it in Rust would be insurmountable. Plus, other comments I've seen suggest that you can actually tune the size of the C zstd library, although I'm not sure if the Rust bindings expose that.
> While it's true that a kitchen-sink version of libzstd does weigh in at 1.5MB, the build is easily configurable and much of that can be shed. If Chromium is just going to ship the decoder, a full build of the decoder is 212kB (162kB stripped, 71kB gzipped). A size-optimized version of that is 99kB (62kB stripped, 26kB gzipped). As of recently, we had someone from the busybox project working to strip that down even more, so we very well may be able to improve further on those numbers.
While still larger than 4K, it does seem possible to make much smaller builds.
I can't remember encountering anything that compresses better with gzip than with brotli, so brotli is usually a better choice than gzip to reduce bandwidth, and brotli has been better than zstd on more than a few occasions for me as noted previously. Brotli is very worthwhile, at least for static assets, and choosing to offer only gzip compressed assets to browsers is almost always a waste of bandwidth.
If faster to compress is all you care about, then sure, brotli might not be the best choice.
I'm looking at my static assets now; our largest JS bundle is 6.3MB uncompressed. With brotli at level 9 it takes 400ms to compress down to 1.34mb, and with zstd at level 13 it squeezes down to 1.31mb in 300ms. If I use zstd at level 19, it takes 1.1s to compress, but size is even further reduced to 1.21mb. Decompression time is identical for both (33ms). So, for our static data the edge goes to zstd.
For dynamic data there's no contest. Most of our responses are binary and brotli is not just slow, it's so slow we have to disable it otherwise people assume our services have hung. By far most of the data that leaves our servers is dynamic, so having zstd support in browsers would be a huge win for us because we would spend less time compressing (compared to gzip) and have smaller responses.
I was looking at one particularly pathological case last month where we had a 2.7gb response that was quite sparse. We were spending 30s of server time as part of the request in gzip, the end result being 52mb. With zstd at level 12 it was 5s to compress it down to 41mb. I don't even know how long it would have taken with brotli as I stopped timing after 10 minutes.
I mean, if brotli wasn't working for people it wouldn't be supported by browsers, but I really really wish zstd was an option.