Ubuntu 21.10 Compressing Debian Packages with Zstd

Clickbait reply: Zstd is a huge advance in compression, but not for reasons you think.

Yes it's fast, yes it has a pretty good compression ratio. But it's greatest strength is streaming feature.

It supports sending chunks of data individually decodable but compressed with knowledge of previous state. Why is this so important?

Because small string compression is crap. For data < few kb, simple dictionary substitution does better than zlib.

But with Zstd streaming, compression ratio rises as more data is sent. Game changer for slow channels.

Imagine messenging over 200 bits/sec channel. You need each message to be individually decompressible. But these are short strings so ratio is terrible.

Enter Zstd. If message delivery is reliable, doable with retries + FEC, you make all messages part of a stream. By the time you've sent a couple hundred, youre getting great compression on tiny strings.

This is huge for constrained channels like NASA rover communications and more mundane IoT networks with low rates and power limitations.

You can get even bigger savings with the knowledge of what the other peer has seen in bidirectional channels. You simply compress the data from the other end mixed with yours. If you have equal data in both directions, combining the prior state this way gets you to good compression ratio terriroty twice as fast.

I have no FB connections with those who wrote Zstd, but i believe this feature was a priority design criteria. Maybe for their balloons or some IoT project, who knows. Either way its a huge advance in data compression.

Its roughly equivalent to how we stream video as diffs from last frame interspersed with keyframes. But for general purpose rather than video.

I don't know anyone currently taking advantage of such property. And you will need to strip out frame headers to make good savings. But a fun project for any of you nerds working on low data rate channels.