Hacker News Reader

Comment by cakoose | Hacker News Reader

The first idea -- generating the reverse delta while applying the forward delta -- seem obvious. The second idea is clever, but Chrome did it in 2011: https://news.ycombinator.com/item?id=2576878

Even if the ideas aren't novel, getting this thing built and rolled out for Windows is not at all trivial, so good job there. But calling it "new compression technology" is misleading.

View on HN

For the second idea, updating references when the code moves is more directly used in a newer Chrome patch format: Zucchini [1]. Courgette does a complex process of disassembling, aligning labels, and sending a patch of the disassembly, so the patcher has to do disassembly, patching, and reassembly. Zucchini does a more straightforward disassembly pass to note where the references are, and then pre-patches them based on how it knows the code moved. Compared with Courgette, this involves less intermediate state so it's faster [2], the code is simpler, and because much more is implicit in the code motion the patches tend to be smaller.

That basic technique goes back at least to the TranslateAddress method in Exediff (1999) [3].

I've been meaning to write up an article on this stuff [4], Google doesn't seem interested in publicizing Zucchini themselves, maybe due to the patent kerfuffle around Courgette. Microsoft's document on delta compression [5] covers a lot of this ground.

[1] https://chromium.googlesource.com/chromium/src/+/HEAD/compon...

[2] Some of this can be avoided, I made some changes to Courgette for a significant speed increase here: https://bugzilla.mozilla.org/show_bug.cgi?id=504624#c39

[3] http://robert.muth.org/Papers/1999-exediff.pdf

[4] I did write up a bug to consider Zucchini in Firefox, with patch size comparisons, but ultimately we didn't switch from the simple power of bsdiff: https://bugzilla.mozilla.org/show_bug.cgi?id=1632374

[5] Their system can use info from the pre-linked objects and PDB symbol files for better alignment, I'd played around with seeding alignments like this in bsdiff and Zucchini but I don't recall it giving significant improvement. https://docs.microsoft.com/en-us/previous-versions/bb417345(...

I desperately wish Linux distros used this trick too. I am tired of waiting for updates to download and having to clean out some cache folder that apt uses to free space.

Last time I looked (years ago), Fedora had delta updates - but updates were slow anyway.

I think a lot of this work is being doing in projects like rpm-ostree

Yep ostree is doing delta updates by default. Flatpaks are alo stored as an ostree, so they do as well - even across all runtimes an flatpaks installed on a system! :)

RPMs have delta updates

But at 50 to 100 MBytes/s on a multi Gb/s line I find it better, it's less the download than the time to install that bother me.

Interestingly I found myself using ultra minimal distro vm, and installing nearly nothing on them (qubes os) and just shard the apps: the startup / update of the main gui linux is trivially fast, each vm only has a little subset of apps I care about, and when I trigger a many-vm update they do it in sequence and just update their own subset, meanwhile I can fully use the other vms.

Also I tend to use stable-like distros now like debian and barely see updates coming in anymore, compared to my fedora youth when there was several updates a day, some breaking everything.

> Interestingly I found myself using ultra minimal distro vm, and installing nearly nothing on them (qubes os) and just shard the apps

Isn't this basically the idea behind snaps? Or is that more docker-like?

I cannot remember when a Debian update was slower than a Windows update. And that does not even mention Visual Studio updates, which pull Gigabytes of data. Debian updates are very fast.

Detecting and installing a Debian update is much faster. But as they are much bigger whether this is offset by the download time depends on your bandwidth.

Nowadays the visual studio updates are quite small for my workload with C++ tools.

For what it's worth, Solus uses deltas since 2016. Updates tend to be amazingly fast. Shame that everything else in it is slowly going downhill.

openSUSE (excluding Tumbleweed) had delta RPMs since 2005.

FWIW, RPM based distros can use delta RPMs

That's more an apt thing, dnf can use delta rpms with parallel downloads and automatically cleans unused packages. I'm not sure how well it cleans the cache though.

On my fast home connection dnf is way faster than apt to update my laptop unless it's a really small update, then apt is faster because it refreshes the repos quicker.

I think there are alternatives to apt that can do parallel downloads and delta as well.

Have they improved the way the system figures out its inventory in order to know what patches it needs?

Windows NT (as a usable thing available to the naive general public) didn't exist before Microsoft built it, so it was technically correct to call it new technology technology. In the same vein, this is arguably new compression technology.

The above being said, my reaction to the article's key point was "you've been shipping forward and reverse diffs this entire time???"

I guess my point is that it's kinda sad that it's most correct to generally describe things like this like as "new technology", because customers who've never seen something like it before don't apply any distinction between that and the idea having existed for however long. </rant>

Well of course the naive public doesn't make a distinction as it would be completely irrelevant to them. Can't use it now? Then it's a theory. A new thing I can buy today? Now that's innovative technology to them.

I always thought that it was just VMS done poorly

The bits based on VMS (or more specifically the MICA project) were done quite well, it's all the Win32 crud on top which causes the problems.

The NT kernel is a nice design especially the (original) support for different flavors of syscalls to support both Windows and POSIX (and potentially others).

A lot of that got lost over the years.

Reply on news.ycombinator.com