Hacker News Reader

Comment by ggm | Hacker News Reader

If your fs implements block dedupe, no need. But, it's expensive to implement and from what I see no longer recommended on ZFS.

If your fs implements hard links, no need inside one mount point. But you can't hard link cross mount.

Copies can be good or bad. It depends.

View on HN

The behaviour is very different to hardlinks. The clone is a new file that uses the same blocks. When you modify the clone, writes are written to new blocks rather than the old ones.

So if I have a disk image that takes 1,000 blocks, and clone it - I now have two images that total 1,000 blocks. And if I modify one block in one image, I now have two different images that total 1,001 blocks. With a hardlink the modification would affect both images.

The way I visualise the difference, is that a directory entry points to a file entry, and a file entry points to a list of blocks. A hardlink is a new directory entry that points to an existing file entry. a linked clone is a new file entry that points to the same list of blocks.

Dedup on FS works, but hard links are useful only if you (and everyone else using the PC) is aware of the hard link (and never forgets about that). Otherwise it's "I can just mess with this file, I have a copy in that other folder for backup".. until you notice it's not an actual copy.

It's better to assume all files are potentially links that must be explicitly copied before any destructive operation. Software should be designed to always produce new copies by default and overwrite files only when explicitly requested.

> But, it's expensive to implement and from what I see no longer recommended on ZFS.

Btrfs (and more recently also XFS) supports on-demand, batch dedup which doesn't have any of the downsides of ZFS's always-active dedup. This is an implementation issue, not a categorical problem of block (or extent) dedup.

Btrfs is like an almost perfect file system. If only it had proper stable support for software RAID. It's weird that it still doesn't have it, I thought enterprises cared about that.

It has proper stable support for many kinds of software RAID: https://btrfs.readthedocs.io/en/latest/Status.html

Except RAID6 which is the one I want. From your link:

https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5...

Also, reading stuff like "mostly OK" in a file system doesn't inspire confidence.

I'd like copy-on-write hardlinks, or deduplicated blocks, that are made automatically on a schedule as to not incur the constant overhead of dedup, but still reap some of the benefits for archival.

Reflinks are basically CoW hardlinks — supported on Linux by btrfs, XFS, and I've seen some effort to bring it to ZFS.

DragonFlyBSD's HAMMER filesystem supports on-demand/scheduled dedup using the hammer(8) command, with configurable memory and runtime limits.

https://manpages.debian.org/testing/duperemove/duperemove.8....

Sweet, thanks.

NTFS deduplication works this way on Windows server since 2012, but I don’t think any of the Linux NTFS drivers support NTFS dedupe correctly.

Reply on news.ycombinator.com