Zstandard really is incredible. Anywhere I would’ve previously compressed things with gzip I now universally prefer to use Zstandard instead. It’s a shame it’s not supported as a HTTP Content-Encoding in web browsers.
https://github.com/facebook/zstd#the-case-for-small-data-com...
zStd also supports pre-trained dictionaries, it is particularly good for specific small JSON, for example. It won't be that helpful for relatively large HTML files.
https://quixdb.github.io/squash-benchmark/unstable/?dataset=...
Perhaps, this common misunderstanding is due to the fact the slowest level (-11) is the default one in the brotli command line utility?
It's also faster than gzip, and gzip was used for "just in time" compression on the web for ages, so I don't see how brotli can be a problem for this use case. (I assume by "encrypt" you mean "compress".)
The thinking seems to be due to how CPU intensive it is, Brotil is not favoured for on-the-fly compression[2].
If you're really desperate for it though, you could try [this extension][3].
[1] https://caddyserver.com/docs/caddyfile/directives/file_serve...
Other webservers support it, mainly Nginx. But since Caddy is written in go, they need a native go port which doesn't exist yet.
Yeah, hence my "really desperate" qualifier. I probably should have been more explicit.
Common reason being someone forgot to check how many bytes read() actually returned and just assume it filled the buffer.
Robust data formats have double protection against this class of bugs with unambiguous SOF and EOF boundaries that the consumer can assert. I guess a sanity-byte once every X Megabytes wouldn’t hurt either.
I still don't understand the use case limitation, or exactly where this is a potential issue. Why is the data being interrupted? Is this some kind of raw UDP stream use case?