It might also be a little unintuitive that modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again.
[0] https://www.truenas.com/docs/references/zfsdeduplication/
Of course, that is true of most filesystems.
I believe as soon as you change a single bite you get a complete copy that’s your own.
And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.
As far as I understand, it works like a reflink feature in the modern linux FSs. If so, thats really cool, and thats also a bit better than the zfs's snapshots. Iam newbie on macos, but it looks amazing
However APFS gives you a number of space related foot-guns if you want. You can overcommit partitions, for example.
It also means if you have 30 GB of files on disk that could take up anywhere from a few hundred K to 30 GB of actual data depending on how many dupes you have.
It’s a crazy world, but it provides some nice features.
I think it stores a delta:
https://en.wikipedia.org/wiki/Apple_File_System?wprov=sfti1#...
If we only have two files, A and its duplicate B with some changes as a diff, this works pretty well. Even if the user deletes A, the OS could just apply the diff to the file on disk, unlink A, and assign B to that file.
But if we have A and two different diffs B1 and B2, then try to delete A, it gets a little murkier. Either you do the above process and recalculate the diff for B2 to make it a diff of B1; or you keep the original A floating around on disk, not linked to any file.
Similarly, if you try to modify A, you'd need to recalculate the diffs for all the duplicates. Alternatively, you could do version tracking and have the duplicate's diffs be on a specific version of A. Then every file would have a chain of diffs stretching back to the original content of the file. Complex but could be useful.
It's certainly an interesting concept but might be more trouble than it's worth.
https://www.vastdata.com/blog/breaking-data-reduction-trade-...
Identifying similar blocks and, maybe sub-rechunking isn’t something I’ve ever considered.
https://www.microsoft.com/en-us/download/details.aspx?id=397...
ZFS is essentially an object store database at one layer; the checksum-hash deduplication table is an object like any other (file, metadata, bookmarks, ...). There is one deduplication table per pool, shared among all its datasets/volumes.
On reads, one does not have to consult the dedup table.
The mechanism was fairly easy to add. And for highly-deduplicatable data that is streaming-write-once-into-quiescent-pool-and-never-modify-or-delete-what's-written-into-a-deduplicated-dataset-or-volume, it was a reasonable mechanism.
In other applications, the deduplication table would tend to grow and spread out, requiring extra seeks for practically every new write into a deduplicated dataset or volume, even if it's just to increment or decrement the refcount for a record.
Destroying a deduplicated dataset has to decrement all its refcounts (and remove entries from the table where it's the only reference), and if your table cannot all fit in ram, the additional IOPS onto spinning media hurt, often very badly. People experimenting with deduplication and who wanted to back out after running into performance issues for typical workloads sometimes determined it was much MUCH faster to destroy the entire pool and restore from backups, rather than wait for a "zfs destroy" on a set of deduplicated snapshots/datasets/volumes to complete.