Varnish Cache and Brotli compression | Hacker News Reader

Varnish Cache and Brotli compression

info.varnish-software.com

27 pointswolfeel4 comments

If it can do gzip, it should be able to do brotli as the HTTP-header "handshake" for both is identical, just use "br" instead of "gzip".

I've been seeing a lot of threads about Brotli lately and having experimented with it thoroughly at work (large social network let's say) and having ramped it up to 100% on production, I would say the benefit over Gzip is marginal and often not worth the increase in latency due to the compression taking longer offsetting your benefits. Of course you can compress once and cache but then there is the cost of storing multiple compressed copies.

SDCH is honestly a lot more promising and we are seeing notable improvements in page load times, on both dynamic and static content. That's ramped at around 10% in production and if things go as expected we will be at a 100% in production rather soon, and write a blog post on it. So far we are seeing 10% improvement in page load times in the United States, and 30% in India, Bangladesh, Vietnam etc. and compressing the content about 50% more compared to just gzip when we are using sdch+gzip. This is all achieved with a dictionary smaller than 700kB.

For every gzip setting, it should be possible to find a brotli setting that is both faster and more dense than your chosen gzip setting, i.e., better in every major aspect: computational resource cost, latency in computation and user observed latency.

If you want to try SDCH and already have brotli, why not brotli+SDCH (brotli sandwich) instead of gzip+SDCH?

Also https://twitter.com/ericlaw/status/690761954915328000 is an interesting observation

We perform SDCH and Brotli on the reverse proxy that's in front of our "front-end" origins. Since some clients at that level need to be able to perform transformations they can only perform on either non-encoded or gzip-encoded responses, we can't have Brotli happen before then. At that point we can either take the cost of inflating the gzipped response and then doing Brotli or just sending it away gzipped. That's why we have a higher threshold for the benefits we would need to get from Brotli before using it more broadly.

We are compressing our SDCH dictionaries with "br" as well. Brotli is great for static content if you know that the user agent that would frequently request that resource supports it. We also have our internal "massively" multi-threaded dictionary generator that we will want to open-source with our plugin.

And we have indeed experimented with all those combinations. Here are some results:

    Using the dictionary generated on <Some popular page>:
    Dictionary generation parameters: [128 iterations per pair]
                                                   [12 threads]
    
               sdch+brotli vs. identity              92.9072%
                 sdch+gzip vs. identity              92.7303%
                    brotli vs. identity              80.0409%
                   sdch+brotli vs. gzip              72.0629%
                     sdch+gzip vs. gzip              71.3661%
                 sdch+brotli vs. brotli              64.4635%
                        brotli vs. gzip              21.3848%
              sdch+brotli vs. sdch+gzip              2.43356%
    
    Using the dictionary generated on <Some popular page>:
    Dictionary generation parameters: [1024 iterations per pair]  <-- more
                                                    [12 threads]      iter
    
               sdch+brotli vs. identity              94.2345%
                 sdch+gzip vs. identity              94.2343%
                    brotli vs. identity              80.0487%
                   sdch+brotli vs. gzip              77.2907%
                     sdch+gzip vs. gzip              77.2898%
                 sdch+brotli vs. brotli              71.1021%
                        brotli vs. gzip              21.4154%
              sdch+brotli vs. sdch+gzip           0.00399263%  <-----

High iteration SDCH dictionary generation eats Brotli's lunch.

This sounds extremely interesting. How do you generate the SDHC dictionary ?

Would it be interesting to compare your results with Zstandard using the same SDHC ? It seems to have a strong dictionary mode.

Reply on news.ycombinator.com