Hacker News Reader

Comment by josephg | Hacker News Reader

The only downside is that zstd has a (relatively) large binary size. I evaluated using it for one of my projects, but it’s a massive library. The zstd-sys crate is 714kb, and zstd has no native rust port presumably because of its size.

I ended up using lz4 instead for my project. Lz4 only added 4kb to my compiled wasm size for an encoder and decoder. And it’s native rust (which makes wasm compilation effortless). Lz4 is a no-brainer to include.

The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.

View on HN

According to a comment in the Chromium issue, it can be configured/stripped down quite a bit:

> While it's true that a kitchen-sink version of libzstd does weigh in at 1.5MB, the build is easily configurable and much of that can be shed. If Chromium is just going to ship the decoder, a full build of the decoder is 212kB (162kB stripped, 71kB gzipped). A size-optimized version of that is 99kB (62kB stripped, 26kB gzipped). As of recently, we had someone from the busybox project working to strip that down even more, so we very well may be able to improve further on those numbers.

While still larger than 4K, it does seem possible to make much smaller builds.

There is a reasonably feature complete implementation of Zstd for Go: https://github.com/klauspost/compress/tree/master/zstd

It may not offer the same API 1:1, but it has no interoperability issues that I've encountered. I think no one has bothered to implement it in Rust because most use cases don't mind the added bloat you're talking about, not because implementing it in Rust would be insurmountable. Plus, other comments I've seen suggest that you can actually tune the size of the C zstd library, although I'm not sure if the Rust bindings expose that.

> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.

lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194

There's also LZ4_HC.

LZ4 (which includes the HC encoder) doesn't have an entropy encoder stage, so it's fairly limited in what compression levels it can achieve.

I feel that is a valid argument for server-side resources not supporting it, but browsers are already hundreds of MB (if not multiple GB) in size.

Exactly, and that couple hundred KB in the browser could pay for itself in terms of bandwidth saved in almost no time over gzip/br.

Reply on news.ycombinator.com