‘You can't use Brotli for dynamic content’ and other misconceptions
certsimple.com
certsimple.com
One of our devs just returned from a conference super excited about brotli. I'm the ops guy and want to try to support excitement, so I spent the better part of 2 days setting up an experiment, but I was skeptical. I'd been keeping my eye on brotli for years.
I first went out and looked for some existing evaluations of brotli, and found years old write-ups, one from Cloudflare from 2015, and another but I can't remember the details there. They both concluded that brotli wasn't ready yet, but they were also old...
Our stack uses Apache on Ubuntu 16.04, and the first stumbling block was that the Apache version on 16.04 is just a bit before the mod_brotli was added. So I used Ansible to spin up an 18.04 web server which did have the right version of Apache. But, apache is built without brotli support. So I ended up having to build my own custom packages, which went fairly smoothly.
We have one particular asset that right now we would like to speed up. It is a big JSON data structure. The dev did some testing and found that his 350K data-set (with gzip) ended up as 320K with brotli.
Maintaining our own custom Apache packages, and possibly having to upgrade our entire stack from 16.04 to 18.04, plus resolve known problems with OS packages that we've seen in 18.04, for a <10% compression gain, for our cherry-picked "best example for brotli wins"?
Our decision was that if we could have just turned it on via changes to the Apache configs, we would have done it. But as it is, it's not worth it for us to do.
YMMV.
We didn't do any before/after testing to see how much data it saved our users but the small tests we've done in the office showed around a 2-5% size reduction for our static assets which was great to see for all the effort of toggling a switch.
(I used to work on mod_pagespeed)
I know how to build my stack, it's encoded in Debian packages all around the Internet.
Another alternative is exploring h2o/caddy as a web server replacement.
Brotli is fine if you can precompress static files on a prep server or desktop.
It's unprofessional and unscientific to evaluate compression codecs without measuring how much CPU and memory they use. Brotli is a disappointment, coming 20 years after gzip. In 20 years we should expect something better than Brotli, something that is better optimized for modern CPUs, vectorized by default, and with a much better dictionary.
However, zstd doesn't have the browser support that brotli does. So you can't easily replace brotli with zstd for web use.
Timeline: Zstandard was released August 31, 2016. By that point, Firefox, Chrome, and Opera already had Brotli support. Safari and Edge got it shortly afterwards.
We serve small to medium sized businesses for our customer base in NZ.
Although I am not surprised: the Edge UI is buggy and the browser engine has its problems with sites too. And we have put a lot of work into ensuring Edge works properly for our clients.
Original post below.
>
Author here. This was submitted with:
'You can't use Brotli for dynamic content' and other horses**t (2018)
The title as it stands now is meaningless, because it applies equally to nearly any pair of compression algorithms.
> Brotli can compress faster than gzip and still produce smaller files
Huh?
> For dynamic content, we'll use 4, which still produces smaller responses but takes less time to compress than gzip or brotli on a higher setting.
Oh if you compare to gzip max. But who runs gzip max? Gzip 6 gives decent low cost compression. Gzip 9 is barely smaller but much slower.
Algorithms have their sweet spots. Brotli (small), LZMA (fast), gzip (somewhere in between).
Knowing nothing about my clients, bandwidth, or dynamism, I can turn on gzip without really a second thought.
What puzzles me though... How much of this 1.1MB is really needed? You can do form validation with HTML5. I don't see any other candidate on their front page. So instead of getting a 14% advantage, what about pushing it to 100% or so?
If you want a pleasant experience, you can’t use HTML form validation by itself. You need some added JavaScript.
(I am not defending over a megabyte of JavaScript. I like to do such things from scratch and optimise for size, and all the functionality that I see on that page looks like it’d comfortably fit in under ten kilobytes of JavaScript. The CSS can probably also be decimated. But these things only happen that way if you value such performance, and few do—perhaps justifiably, perhaps not.)
CertSimple is a full dashboard and application that does a lot of crypto in the browser which requires quite a lot of code.
Sure, there are probably gains to be had with code splitting and lazy loading, but those get significantly more complex and can cause issues for codebases that aren't setup to take advantage of it from the start.
And improvements are improvements. This was a blog post showing you how and why to implement brotli, not a request for how to improve page loading speed for their application specifically.
> the front page of CertSimple is 242 K gzipped, and would be 1.1MB uncompressed!
Of course the full app is a completely different thing.
Author here: no. The app is the front page.
/blog is a seperate AMP based site and tiny. Search 'discify' in this thread for a breakdown of what's in the module.
Mike, I’m afraid you have bigger problems than inefficient compression...
We use https://github.com/131/discify to analyse this and should be able to cut down a little further once the URL spec is more widely implemented.
animation:-amp-start 8s steps(1,end) 0s 1 normal bothI don't know specifically what amp.js is doing there with the animation but since the entire purpose of AMP is performance, I'm sure there's a solid motivation behind it.
The snippet I posted is a small bit of inline CSS at the top of your page that hides all page content for 8s. There is no JS involved, and the mechanism is quite clear if you know CSS.
> the entire purpose of AMP is performance
I'm afraid you are sorely mistaken. (See the comments on any AMP-related HN submission if this is news to you).
> I'm sure there's a solid motivation behind it.
What makes you so sure? The motivation is to artificially degrade the experience for any users that choose to block 3rd-party AMP tracking scripts. There is no other reason behind it.
> I don't know specifically what amp.js is doing
Normally, one can overlook people throwing random 3rd-party scripts into their blog with no clue what they do, but for someone blogging about using brotli and commenting about applying discify, this kind of wilful ignorance is a little ironic. Particularly after someone points out that there is an 8s delay in your page load.
No, the inline CSS is added by amp.js.
> > the entire purpose of AMP is performance
> I'm afraid you are sorely mistaken (See the comments on any AMP-related HN submission if this is news to you).
I wrote a bunch of those comments. Now Malte is allowing people to host AMP content on their own domains I trust it a little more.
> The motivation is to artificially degrade the experience for any users that choose to block 3rd-party AMP tracking scripts.
AFAIK amp.js doesn't do any tracking on it's own - you need an analytics module to do that. Do you know differently?
> > I don't know specifically what amp.js is doing > this kind of wilful ignorance is a little ironic
Not really. Do you know your CPU microcode well or would you consider yourself wilfully ignorant? Perhaps rather than being wilfully ignorant, I choose to focus my time on what matters to my customers (with a side of arguing on Hacker News).
It definitely isn't. It's in the raw source, sans-JS. Whomever developed the blog theme placed it there explicitly. (it is a snippet given to you to copy paste while implement amp, but it serves no purpose other than to degrade performance for non-amp users).
> Malte is allowing people to host AMP content on their own domains I trust it a little more.
That's not how "AMP cache" hosting works. AMP is a system whereby the site source is crawled and rehosted on the server of an indexer. You can host it on your own domain to make yourself feel a little better, but the only people who see that copy will be those to whom you give that direct link (e.g. HN readers). Indexers are free to rehost your AMP content on their own "AMP cache". i.e. it's still going to be served from https://www.google.com/amp/s/certsimple.com/blog/nginx-brotl... for any mobile search traffic coming from Google (note the link is referer-dependent).
> AFAIK amp.js doesn't do any tracking on it's own - you need an analytics module to do that. Do you know differently?
You need an analytics module to avail of an analytics service to do your own tracking. There's nothing to indicate that Google do no internal tracking of their own.
> Do you know your CPU microcode well or would you consider yourself wilfully ignorant?
The intent of my comment was to point out that being aware of this issue involves a relatively low level of knowledge/investigation/interest. So low in fact that in my very first comment, I posted the entire 53 character sourcecode of the issue. Also, CPU microcode changes aren't causing 8s page load delays for me.
From that: AMP HTML documents must contain the following boilerplate in their head tag.
> That's not how "AMP cache" hosting works.
With prefetching my understanding (from what's been written publically) is that content will be allowed to be hosted on your own domain. Ie, your domain in the address bar. Are you saying otherwise?
> There's nothing to indicate that Google do no internal tracking of their own.
OK so you just disagree on where the burden of proof lies.
> The intent of my comment was to point out that being aware of this issue involves a relatively low level of knowledge/investigation/interest.
You think it causes an 8 second delay. It doesn't. The page loads unbrowsercached in less than a second. The low level of knowledge/investigation/interest still seems to be above your own.
No. As I said, you can host it where you want. You can visit that domain and see your content. But Google are still going to rehost your content on their own server and direct their users to their copy, on their domain.
> You think it causes an 8 second delay. It doesn't.
Apologies, I assumed you'd have some understanding of frontend technologies when I posted the code above, so I'll explain how it works:
- you have an inline CSS snippet that hides all page content for 8s (there is no reason for this). This is executed immediately.
- you then have your amp.js file, pulled from Google's server, which should download and parse relatively fast (especially if you've put the script tag in the document head, as is recommended)
- once amp.js is loaded and is parsed, it enables some further inline CSS which overrides the first inline CSS snippet, unhiding all page content instantaneously
As I said above, the 8s delay is for non-AMP users; anyone not loading the amp.js resource from Google's servers, or anyone for whom the resource fails to load for any reason.
As long as my domain is in the address bar, and the content isn't modified - which I understand will be the case - I don't care.
> As I said above, the 8s delay is for non-AMP users;
No, that's not what you 'said' (wrote) above.
You wrote:
>> removing the artificial 8s delay on the CertSimple blog might be another line of inquiry.
You wrote:
>> The snippet I posted is a small bit of inline CSS at the top of your page that hides all page content for 8s.
Implying there was an 8 second delay that normal users would see.
Even now you're jumping through hoops to say 'non AMP users' where in reality you mean a tiny niche of users with broken network connectivity or who deliberate disable JavaScript, etc.
Thanks for wasting my time. If you'd have said what you honestly knew "I dislike this as it won't show the site if users can't load the JavaScript" I would have said "fine I don't care". Instead you pretended something was actually wrong and I wasted time communicating with you and being insulted.
Please do not communicate with me again. This is why HN needs a block button.
Perf is always fun to optimise, but we're got features customers want that we need to work on first!
One of the best benchmark about web compression including brotli + different gzip compatible libraries:
Compression ratios give information about the contents of a message. Hence, it seems to make sense to only compress static content from a security standpoint.
“Maximum” is good because it handles a common use case. When you’re running a CLI tool, you’re much more likely to be processing static assets which are served many times. So “maximum” is a good guess of what the user wants.
I'd especially like to use Brotli because it has good support in modern browsers (pretty much everything except IE11) and would be good replacement for gzip (which should be disabled for HTTPS).
?
Please explain.
There are workarounds / mitigations with various effectiveness, but disabling compression when TLS is used is the simple way to prevent the attack.
[0] https://tools.ietf.org/html/draft-pironti-tls-length-hiding-...