In general, compression works much better the more data it has to work with. When you're talking about one or two KiB files, the overheads are going to eat into a lot of your gains. It's definitely still worthwhile, but I can see how a custom packed binary format could easily outperform a compressed text format at these sizes.

The 7 KiB SVG is almost certainly too large for that image, but I guess it won't get much smaller than 1 or 2 KiB in that case. Not sure how SVG-specific brotli is. Wasn't the dictionary mostly HTML, which would save pretty much nothing for SVG?