Hacker News Reader

Comment by ryan29 | Hacker News Reader

As good as ZFS is, are people relying on ZFS replication for all of their redundancy and backups? Is that a good idea in terms of risk mitigation?

When I set up backups for my home lab server, I opted to go with Borg + Borgmatic + BorgBase for backups. I already use Sanoid + Syncoid for local snapshots and replication, so I was hesitant to rely on ZFS replication for offsite backups too.

Instead, I use Borgmatic hooks to link the most current Sanoid snapshot to a well known location, run my backups to BorgBase, and delete the link. I'm still relying on ZFS snapshots, but the backups, versioning, pruning, etc. are a completely different system than the local replication.

I'm also not a fan of the full + incremental strategy for backups since I think it encourages long increment chains.

View on HN

> Is that a good idea in terms of risk mitigation?

It is definitely not. Snapshots + replication is a great 99.9% "hot data" backup, but they don't guard against administrative error or massive pwnage (think ransomware). At least three copies, at least two locations, at least two different systems (media/backup device vendors/operating systems/etc.) is the full package, so to speak.

EDIT: I don't want to crap on this project as it serves a great use case. But any filesystem-based replication should be backed up by a logical copy, e.g. a tar or a zip file or just an "aws s3 cp -r". I have seen even checksummed file systems replicate data incorrectly due to bugs, unlucky bit-flips, ECC errors, CPUs literally taking the wrong branch. And remember, at the base of everything, this stuff is written in C!

If the AWS IAM rules have no overwrite and delete permissions then both admin and ransomware aren't really an issue are they?

Obviously putting all your buckets on AWS performing correctly is a potential risk by itself, but it's a different kind of risk.

rsync.net allows you to send zfs send your own filesystem, and they have provisions to make the data append-only so you can't fall victim to ransomware.

Of course it doesn't protect against bugs in zfs itself, so it's still good to use another software depending on how critical your data is but I'd say for a very large majority of cases it should be enough

> they don't guard against administrative error or massive pwnage (think ransomware).

It does if you fully automate it and also invert control and think of your backup servers as a place to deposit stuff which is then rotated out into immutable snapshots - this is how rsync.net works, but I couldn't use them due to location requirements so I built my own based on ZoL.

> any filesystem-based replication should be backed up by a logical copy, e.g. a tar or a zip file or just an "aws s3 cp -r". I have seen even checksummed file systems replicate data incorrectly due to bugs, unlucky bit-flips, ECC errors, CPUs literally taking the wrong branch

This is literally what ZFS was designed for, I would agree with you if it was any other file system, but ZFS is not any other file system - of course you do need to throw enough redundant hardware at it for it to be able to cope with this stuff though, a mirrored pair at minimum.

I certainly know what ZFS was designed for. I've been working with replicated checksummed file systems for decades, including WAFL, NetApp's file system, which ZFS was "inspired" by (see the Sun vs NetApp lawsuit). They had both block- and sector-based checksums in 2002, albeit designed for RAID-4. I wholeheartedly endorse checksummed file systems for maximum data integrity.

ZFS will not protect you (by default) against bit flips in memory (please use ECC) or kernel bugs. If the disk buffer is corrupted in the kernel, on its way to the file system layer for dispatch, ZFS will happily checksum and perfectly store the bad data. Checksums are like Rust's borrow checker: a very good idea that makes your data safer, but not a panacea that prevents all errors.

Finally, as with all complex file systems, ZFS is notorious for difficulty in recovering from multi-bit errors, often requiring "pool surgery" and multi-day disk scrubs to remount the file system. Perhaps your data is eventually recoverable, but do you have a week to finish the scrub to get it back online?

If any this sounds unlikely to you, I assure you, it is! But it can happen, I've seen it happen, and for things that really matter, one of your backups should always always always be logical.

> ZFS will not protect you (by default) against bit flips in memory (please use ECC) or kernel bugs. If the disk buffer is corrupted in the kernel, on its way to the file system layer for dispatch, ZFS will happily checksum and perfectly store the bad data.

That's all very well but I don't see how your suggestions of doing archive backups make this any less of a problem, you are just operating on a higher level layer with the same problems. I'm also fairly sure most servers are built with ECC these days, I know my IaaS provider uses it.

> Finally, as with all complex file systems, ZFS is notorious for difficulty in recovering from multi-bit errors, often requiring "pool surgery" and multi-day disk scrubs to remount the file system. Perhaps your data is eventually recoverable, but do you have a week to finish the scrub to get it back online?

I cannot claim to have extensive ZFS war stories but I have had errors on a bad block device in a mirrored pool, but with this type of pool it does not affect mountability or performance. The only annoying performance issue affecting mountability I've had so far was when accidentally filling up a pool that had dedup enabled, since it struggles to function in the reserve area.

Wouldn't a tar / zip / whatever suffer from the same exact problem of making a corrupted backup if the filesystem returns the wrong data?

Definitely, if the file was corrupted before entering ZFS's checksum regime—it won't help at all. However there are a few scenarios where making a logical copy, regardless of archive format, helps.

(1) By reading the files you are "tickling" the file system to re-validate the checksums of the blocks retrieved. Note that ZFS doesn't by default do this for data in memory, and with modern memory sizes, buffer pages can stay around for a very long time. Of course if your memory pool is large enough and your data set small enough, your backup might not actually force disk access. You can be really paranoid and flush the page cache before the backup starts. (This used to be "echo 1 > /proc/sys/vm/drop_caches" for Linux, but kernel code changes quickly, so don't take my word on it.) This will make the backup job preen your original data and flush out any disk corruption which occurred between reads—quite useful if a lot of your data is cold. However it's typically a big performance hit and might be a little too paranoid even for me.

(2) The act of reading the files' metadata and translating it into kernel data structures exercises quite a bit more code than copying a list of blocks onto a wire. I don't know ZFS's replication protocol[a], and the developers aren't dummies, so I'm sure they've thought of a lot of corner cases. But I have definitely seen filesystem-level replication copy some subtle underlying metadata issues which weren't detected until the files were read logically. Typically this is caused by code errors rather than hardware failure. The problem with these kinds of bugs is they've often rampaged behind the scenes for a few days by the time the first error shows up; logical reads will flush them out a lot faster.

(3) The act of reading and translating the data to a new format will get your hands on it, and a lot of subtle systemic problems (e.g. application failures) can reveal themselves in the process. Filesystem replication is very seamless and quiet when it's working, and it's easy to lose track of what's actually going on. E.g. I once noticed my backups were broken because the nightly spike of bandwidth on our network graphs got a lot skinnier. There were notifications and alarms everywhere, none of which triggered. It turned out the database servers were writing to the wrong drive (shadow mounts). Replication of null data is, of course, not an error.

[a] I would love to read a non-Oracle study of ZFS replication, if somebody has one—Google does not provide at the moment.

A long time ago when I designed a zfs based backup + offsite-backup solution (having the luxury of 2 backup levels) I intentionally used ZFS replication between the data and backup but then used file-based replication between the backup and offsite-backup. I also ensured that you couldn't (at least easily) directly access or influence the snapshot state of all 3 locations by starting at one of the hosts. The offsite-backup location also had it's own point-in-time snapshots.

The reason is that (a) zfs send/recv is very fast and can be done on an hourly (or even faster) basis even with billions of files, but file-based replication (e.g. rsync or rclone) is typically slower with large file counts. We can afford to have a slower pipeline to the offsite-storage.

(b) I wanted to prevent some kind of ZFS bug in send/recv handling accidentally transferring corruption from the original data store to both backups

(c) Depending on the zfs send/recv flags that are used, it can remove snapshots not on the source host. So there is scope for both accidental and malicious deletion of all snapshots depending on the circumstance.

(d) You don't want all files being removed on the source to remove all copies in either the backup or offsite-backup location. That is the reason the offsite-backup had it's own point-in-time snapshots so that if all of the files are deleted from the source + backup, when the offsite-backup replicates that state and removes all of it's files, you can still access the files from the previous point-in-time snapshot.

(e) You don't want a hacker that penetrates 1 of the 3 storage locations to be able to cause or directly access both of the other 2 locations. This is commonly possible because SSH keys are used to allow zfs send/recv or rsync. Pivoting from a production storage to all backups (and destroying them) has happened in some high profile malicious cases, e.g. Distributed IT.. At a basic level, this means for example your backup server can't directly co-ordinate both the local and offsite backups via SSH otherwise it may have direct access to both environments and if the backup server is compromised.

It seems to me you could get away with it being most of your redundancy, especially on systems that aren't mission critical. I mean, between CoW and the Merkle Tree of checksums going on the machine would have to literally random walk into a hash collision for the data to be unrecoverable.

Reply on news.ycombinator.com