Hacker News Reader

Comment by terrelln | Hacker News Reader

Disclaimer: I'm a maintainer of zstd, so I'm biased.

Brotli dominates HTTP compression. Zstd just got its RFC approved a few months ago, but Brotli has been present in browsers for years.

However, zstd is more widely adopted everywhere else, especially in lower level systems. Zstd is present in compressed file systems (BtrFS, SquashFS, and ZFS), Mercurial, databases, caches, tar, libarchive, package managers (rpm and soon pacman). There is a pretty complete list here https://facebook.github.io/zstd/.

Again, I'm biased because I know almost everywhere where zstd is deployed, but not everywhere that Brotli is.

View on HN

I'm also biased, as I'm the author of Brotli. With Brotli you get about 5 % more density. Brotli decompresses about 500 MB/s while Zstd decompresses about 700 MB/s. Typical web page is 100 kB, so you need to wait 200 us for decompression. For zstd, you'd only need to wait 140 us.

That 60 us saving comes with a cost: you'll be transferring 5 % more bytes, which can cost you a hundreds of milliseconds. Brotli is also more streaming, so you get your bytes our earlier during the data transfer. This allows for pages to be rendered with partial content and further fetches to be issued earlier.

Zstd supporters have used comparisons against brotli where they compare a small window brotli against large window zstd. This makes it seem like zstd can compete in density, too, but that is just apples to oranges comparisons.

Brotli is great at compressing static web content, zstd without a dictionary is unlikely to outperform it. For static content you'd probably rather save 5% of space over some decompression costs, since Brotli decompression is fast enough.

Zstd has an advantage if you don't have the CPU to compress at the maximum level, since zstd is generally faster than Brotli at the lower levels.

Even still, for web compression, Brotli has the advantage of already being present in the browsers, so you're betting off using Brotli for web compression as it stands today.

For encoding there shouldn't be a format specific difference. If there is, it is based on immaturity of the implementations. There were stages when brotli:0 used to be faster to compress than any setting in zstd, but now they have played catch-up and are the leader.

Zstd can be significantly slower in medium levels -- you just need not to be tricked to apply compressors at different window sizes. Zstd changes window sizes under the hood. With brotli you need to explicitly decide about your decoding resource use.

If you use the same window size and aim for the same density of compression, brotli actually tends to compress faster in the middle qualities, too.

Excited for Zstd getting into browsers, Brotli is great for static assets, but it doesn't really outperform gzip by much at dynamic compression scenarios.

If you made your conclusion's from https://blog.cloudflare.com/results-experimenting-brotli/ be warned that both of it's results and conclusions are flawed, because of a significant methodology flaw.

You just cannot compare two things with a commutative function, and that blog post is based on an assumption that you can. Math just doesn't work like that.

Brotli's fastest compression modes are 3-5x faster than zlib's fastest modes. For every gzip quality setting there is a brotli setting that is both faster and more dense than that gzip setting.

We used https://quixdb.github.io/squash-benchmark/ mostly. It's not that Brotli wasn't a winner compared to our gzip, it just didn't outperform it substantially and came with challenges integrating it with our product for supporting dynamic payloads. Meanwhile static assets just need brotli support at build time.

While it is a great benchmark, that is about 3 year old data. The brotli quality 1 was moved to be quality 2, and two new levels have been added. Everything has been sped up and a few levels (possibly 5-10) have got a ~5 % density boost from improved context modeling. Level 10 has been added (about 2-3x faster than level 11) -- in squash benchmark you still see the initial behavior where level 10 copied level 11 performance.

With the zstd support in pacman, arch will soon switch all packages over to zstd.

Nitpick: it's actually in libarchive, pacman/makepkg doesn't really care about the format.

Out of curiosity: Why doesn't pacman just use HTTP's built-in compression? It could cache packages in gzipped form, but there's no reason to re-compress them over the wire.

What would be gain?

Because mirrors aren't guaranteed to use HTTP gzip compression and also gzip compresses much worse than xz.

Reply on news.ycombinator.com