Really fun to read article, but could someone give me a couple examples of what kind of system you'd use this with? I guess I'm so used to S3 now that I've forgotten what it's like to manage your own files.
You can do neat things like: create a base filesystem, create a snapshot of it (i.e. "freeze" the filesystem in time), send the snapshot to another system (or over the network for S3 storage, etc), continue modifying files, create another snapshot on top of your first one, now you can send/backup the new snapshot which is only a diff of the original snapshot.
Many other systems besides ZFS support snapshots/copy-on-write layers, but they are mostly hacks and your performance will drop considerably during the "snapshot existing" time. ZFS is different because the snapshot mechanism doesn't kill your performance (unlike the 18 layer Linux file system organization).
"Sending" a snapshot can either be directly over the network to another system to ingest the snapshot (enabling you to stream a new filesystem/snapshot into existence on a remote machine), or you can gzip the snapshot stream and send it off to backup storage.
You can layer your snapshot diffs as much as you like. As long as you have all the diffs back to the original file system, you can re-create any point-in-time snapshot layer from your backup archive.
Longer details with examples are all over, but a good overview point is: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm...
Also, it's basically the only sane way to run containers in production since all the copy-on-write and snapshot sending isn't an 8 layer deep hack cake. In Solaris, containers are aware of filesystems at the deepest levels and the networking is aware of containers too. All Linux-based container fads over the past 5 years still haven't caught up to Solaris+ZFS+Zones+Crossbow from 6 to 10 years ago.
All hosted e-mail lives under /var/vmail on its own ZFS dataset ("filesystem"). I snapshot this dataset (via cron) several times per hour and ship those snapshots off to another machine in a different datacenter for safekeeping. If anything tragic were to happen to the mail server itself, I've got a pristine copy of all my users' mailboxes I can turn to that will be, at most, roughly 15 minutes out of date.
With S3, I don't believe it has any ability to diff and/or store and transmit diffs - S3 is just a data store.
ZFS replication / rsync is more for the use case where you want to make sure files on multiple sides of the data transfer equation are the same, and can pass diffs back and forth between them.
Basically, if you need and only use S3 and file storage, this isn't want you want at all. The closest analogue is a binary git merge for very very very large files.
rsync is smarter it actually can compare and send parts of modified files. This article talks about new functionality which utilizes zfs snapshots. Now zfs knows better what has changed (for example whether file was renamed or moved) so it can be even faster when replicating the changes.