Hacker News Reader

Comment by terrelln | Hacker News Reader

One of the maintainers of zstd here, AMA

What's the future of binary deltas in Zstd? I saw some work on a github chat, but not sure if it's being seriously planned.

Zstd fully supports binary deltas. If you're computing a delta between `old` and `new` you would do this to round trip it:

zstd --patch-from=old new -c | zstd -d --patch-from=old

Using the library this can be achieved by using the old file as a dictionary. If you have very large files (MB-GB range) you will also want to enable long range mode, otherwise zstd might not be able to find the repetition. --patch-from on the CLI will manage this automatically.

The release notes from zstd-1.4.5 [0] contain some details about the speed/size tradeoff of "patch-from" mode, compared to other delta engines. There is also a wiki page detailing how to use zstd as a delta engine [1].

[0] https://github.com/facebook/zstd/releases/tag/v1.4.5

[1] https://github.com/facebook/zstd/wiki/Zstandard-as-a-patchin...

I don't think you are doing the correct comparison in your article. BSDiff is two steps, the first step is calculating the binary subtraction mappings, and then compressing these mappings with a compressor. Colin Percival used zlib compression in his original bsdiff paper, but the performance of zlib is not great. If you really wanted to compare zstd deltas with bsdiff you need to replace zlib with a modern compressor like regular old zstd.

You're right that a totally fair comparison would use zstd. I didn't personally run these benchmarks, but I believe that the dominate time was computing the binary subtraction mappings, so I would expect similar results.

Also, if you give zstd binary delta a try, we would be super interested in hearing the results you get! Especially around the tradeoff between compressed size, compression time, and decompression time. Opening an issue would be the best way to communicate it.

Reply on news.ycombinator.com