Hacker News Reader

Comment by zorkw4rg | Hacker News Reader

basically the reason I like my filesystems a few decades old and mature, I would not trust zfs or btrfs with anything critical.

View on HN

I have been running ZFS on macOS (OS X), illumos and Solaris for a good 7 years or so now. A major part of the reason I switched over fully, despite some warts, was that I experienced actual and significant data rot from stuff I was carrying forward under XFS (IRIX), HFS, etc. I don't consider my personal stuff to go back that long, but I still have things from 1993 or so that matter to me. I did a review around 2009ish, and found that a number of old files had at some point or another become corrupted, including ones I know for sure weren't in 2004. I'd been following ZFS somewhat since Sun had demoed it, including a depressing spell after Apple almost moved to it then quit, but it was my own personal actual losses that pushed me to move over.

End to end checksumming and other integrity features just plain should be universal at this point. Should have been a decade ago or more in fact. We have so much incredibly important data now that is digital only and nowhere else, and memory and storage have both become very cheap at the general population level. It's shameful that anyone should still be losing data or experiencing anxiety years or even decades down the line. Integrity, basic levels of security/privacy, and flexible, high integrity replication should all be native level features of any data store system. "Decades old" filesystems just plain absolutely do not cut it, no does anything newer that doesn't include those promises at least as options. Bugs are unfortunate, and I hope this prompts ZoL and associated projects under the OpenZFS umbrella to double check their automated unit and stress tests. Sun rightly made a big deal of that on release. Even so, I wholeheartedly believe that ZFS or the like are far better primitives for a data storage scheme then older FS (or many newer ones for that matter).

I relate to your usage scenario, and my requirements are similar, and so are my frustrations with older filesystems: little or placebo protections against silent data corruption, lack of CoW semantics, pain of deduplication.

But on the other hand, over the years, I've heard so much FUD -- or underwhelming rumours that may-or-may-not be FUD -- about both ZFS and btrfs that have made me very reluctant to jump ship, and instead lean on an extensive suite of homemade processes and procedures that badly try to replicate subsets of features I'd expect a modern filesystem to provide. This is likely a Bad Idea, but my trust model right now can better stomach me losing data due to my bug, than someone else's.

I think what I'd like to see is widespread deployments, deployments-by-default, and better, persistent press and/or marketing surrounding *ZFS or brtfs, such that I wouldn't feel like an early adopter when using these filesystems. I'm actually glad Apple secretly wrote a filesystem with some of the same ideas, and deployed it to production on millions of (users') devices in the wild, because it raises the profile of modern filesystems and increases the likelihood that comparable alternatives will see more attention.

I am still somewhat suprised there are still not relatively simple filesystems that don't do this. Make it the one feature it does and does properly. Adding enterprise features will just increase development time (and cost) and add bugs. Basically an online archival filesystem, that's how most people use their home computers.

Requirements:

1. Checksumming on all files

2. Minimise assumptions of ram correctness

3. Disc replication (soft raid 1)

Out of scope:

* Snapshotting

* Subvolumes

* Deduplication

As much as I love subvolumes and snapshotting, I feel like CoW probably adds too much complexity to make it (easily) reliable. Honestly I don't know why turning off CoW on BTRFS disables checksumming, so if anyone can shed some light as to why this is, feel free to point out that my requirements are far more complex than I think.

In-place writes invalidate checksums. It is theoretically possible for an in-place filesystem to have data checksums, but maintaining them would require reading the entire block to regenerate the checksum on every write (I consider reads from cache to be reads). That largely defeats the purpose of doing operations in-place, which is to avoid that overhead. Maybe it would make more sense on btrfs (because it’s metadata structures can become horribly unbalanced) than it usually would on a convention in-place filesystem. However, the code would need to do gymnastics to achieve that without risking subtle reliability issues. That is something that I would not expect many filesystem developers to want to maintain. The btrfs codebase is complex enough (more so than ZFS despite having fewer features) without adding that.

By the way, partial writes from power failures are impossible to correct on in place filesystem designs without writing everything twice. CoW has performance penalties, but not so much as in-place designs when it comes to ensuring integrity. The complexity of doing CoW is also not that bad. It is how virtual memory works in every modern computer system, minus some rare embedded systems and ancient/unikernel designs that lack it. Doing CoW in storage is not a particularly strange thing.

> I don't know why turning off CoW on BTRFS disables checksumming

Because in-place updates means you need to journal data writes, otherwise you'll get checksum errors for aborted writes. (Which classic file systems don't catch)

The recent (~5 years old) batch of block-deduplicating backup tools (attic, bup, borg, restic etc.) would fit the bill to some extent, but all of them were/are more bug-laden than the file systems discussed so far.

Checksumming is like a myth on HN already. No, it's not useful on its own to the user or to the filesystem, but it is useful to the filesystem if it can transparently heal the data. Whole disk replication doesn't address your reliability concerns as much as you think either, doesn't work all that well in general and is a burden to maintain. Doing it properly requires treating the filesystem as a distributed system where disks can join and leave, where everything is sharded, where rebalancing, self-healing, syncing, resyncing is all automatic and blazing fast. And so on.

Nowadays most of the work in storage systems is in distributed systems, and local filesystems are treated as just another unreliable layer.

Hope this gives you some ideas why there are no filesystems like that.

Not particularly. I'm thinking for the home user who just copies the photos from their camera to their home computer. Distributed systems would be great, were it not for the fact that by and large, home connections have abysmal upload speeds.

>As much as I love subvolumes and snapshotting, I feel like CoW probably adds too much complexity to make it (easily) reliable.

Despite this bug, ZFS still seems reliable and does everything you ask. And any reliability you feel is gained by dropped CoW is offset by the reliability lost by being susceptible to power outages corrupting data.

I feel similarly. I've had basically all my data on ZFS/ZoL for 8 years at this point and have not had any problems except occasional system lockups when scrubbing. This has not happened for the last year or so.

For a while it seemed like ZoL was working hard to catch up with the "official" ZFS feature set, but at this point I hope they take it slowly and carefully and really beat on new upgrades.

The more stable you are, the more important it becomes for a FS to not screw it up, since the consequences get larger with a larger user base...

It sounds like you are suggesting that checksums and other integrity features obviate the need to back up your important files.

They're part of the same process, you need the checksums to know your backup is reliable and to tell you when your main files become corrupt. Backups alone aren't enough either.

No, it doesn't work like that, backup is not redundancy. To detect a bad block you first need to read it, you can't just know when a file becomes corrupt. And reading every block is very expensive and doesn't provide predictable guarantees. Instead proper backup has built in redundancy to account for lost blocks and so does proper data storage. Checksums are not visible to users in neither case.

And backup has to be incremental to prevent disasters.

I don't understand how you could have possibly gotten that, particularly given that I very explicitly finished with "that ZFS or the like are far better primitives for a data storage scheme then older FS". Primitives. Other components are still important. But backups in no way whatsoever inherently provide integrity, that's not the threat scenario they deal with. A non-verifying backup (quite typical, especially historically) will cheerfully and unquestioningly backup corrupted data, and can become corrupted itself over time. When you're considering data maintenance over the course of not just years but decades, "oh just backup" isn't at all enough without an integrity story to go with it.

Now of course it is possible to build levels of verification higher up the stack, but the problem with a lot of the ones available to general user is that they either leave holes in various places or they're just plain too much of a PITA (and "PITA" here includes a significant enough level of slow down) or both. It's much easier to do a good job if the FS itself is reasonably trustable and has verification and basic repair built in. FWIW I do use tools like par2 for cold or cool backups of ZFS dumps to targets like tape or Glacier now, but it's still a big help to my sanity to have as much of that dealt with at a low, automated level for hot/warm systems where convenience is critical.

Of course not. But using fss lacking these features the backups end up 'faithfully' (and uselessly) propogating bitrotted files.

Are you still using ext2 then? Ext3 was 2001, and ZFS was 2005. JFS2 is only 19 years old so doesn't quite meet the "decades" requirement.

not parent, but -

ext3 is improved ext2 which is improved minix fs which is improved fat, and all use the 'traditional' static-map-of-blocks approach, with gradual incremental improvements on top of this.. so the general technology 'family' is decades old even if any one implementation is not..

zfs/btrfs are log-structured, which wasn't really feasible from a hardware/performance standpoint until much later (see also LFS), and so represent implementations from a later 'generation' of technologies

I'll give you ext3, certainly as ext2 and ext3 share a codebase. They are wholly unrelated to FAT though, ext and minix both derive from the UFS/FFS branch of file system development which has little, if nothing in common with FAT filesystems.

[edit] The original 8-bit FAT and FFS were both implemented at about the same time, if you consider FFS's beginning to be when I-nodes were moved into cylinder groups. If you consider the Unix "FS" to be the beginning, then it predates FAT by a lot. In any event inode and cluster-chain systems have fairly significant differences.

Also, long-structured/CoW systems have been used as backends for databases for about 40 years now.

ZFS and btrfs are not log structured, but CoW. There are some aspects of ZFS like ZIL and L2ARC that are essentially log structured, but the filesystem itself is not a log structured filesystem.

Using CoW in storage is an idea that occurred after in-place and log structured filesystems had been made as far as I know though.

He's likely referring to XFS, which meets the requirement.

Considering that XFS is currently getting new features en masse, this doesn't sound logical. Except when they mean they are using a still maintained kernel before Linux 4.x.

And just to round it out, UFS2 was first in a release in 2003 in FreeBSD 5.0. So on the BSDs you're stuck with UFS1 at "decades."

Alternatively you can just avoid the bleeding edge versions of any file system.

Yes. I don't think anyone would contend that it's not bad for this to have landed in a stable release, but in fairness, that stable release was about 3 weeks old and ZoL is not yet 1.0. Mistakes like this are unfortunate but they do happen, and that's why many people wait several months before they incorporate new versions.

From the bug report, sounds like ZoL is integrating some new tests, learning how to retrieve orphaned files, and possibly strengthening its hashing mechanisms, so overall this will strengthen ZFS's robustness. Little consolation for those with damaged filesystems, but that's how the cookie crumbles sometimes.

Reply on news.ycombinator.com