Muxfs – a mirroring, checksumming, and self-healing filesystem layer for OpenBSD
sdadams.org
sdadams.org
(1) (And the largest problem) is that it requires stable inodes in order to tie the checksums with the actual files. This means (and it's already stated in the article) you can't copy / move / overwrite any of the underlying files without losing the checksums. (Basically it also removes the possibility of accessing one of the mirrors via NFS, FUSE, or anything that doesn't have stable inodes.)
(2) (Based on my reading of the article) it doesn't seem to hold a "log" or "sequence" to identify which of the two mirrors are ahead or if they are in sync. In case of a disconnect / reconnect you need to manually tell `muxfs` which is the "newer" one (by using a `sync` before being able to mount it).
(I haven't tested it though, I'm running Linux, but I'm quite interested because just last week I thought "why doesn't one implement a FUSE file-system to add checksums and thus prevent bitrot". `muxfs` also adds mirroring.)
OpenBSD also supports extended file attributes, so using them should be possible.
Using extended attributes on Linux or FreeBSD requires a few precautions, because there are still various copying/movement/archiving CLI commands or GUI applications that ignore the extended attributes and also some file systems that do not implement extended attributes, e.g. tmpfs on Linux (which supports only certain kinds of system extended attributes, not those defined by the users) or all not extremely new versions of NFS (only NFSv4 in Linux 5.9 or newer supports xattr, unlike samba, which has supported them for decades, mapping them correctly between different file systems, e.g. XFS on Linux to UFS on FreeBSD), so copying a file via those file systems would lose silently the extended attributes of the files.
The extended file attributes have been introduced in 1989, in HPFS for OS/2 version 1.2, and they have been brought to UNIX in XFS, in 1993.
30 years later, it is annoying to see that there are still some programs which pretend to make file copies or file archives, but which can lose the extended file attributes, without any warnings or errors.
Extended file attributes can be used to implement a large number of useful things, many of them enhancing security, e.g. access-control lists. However, IIRC OpenBSD has chosen to also not implement ACLs.
In any case regardless how useful or not useful extended file attributes are considered to be, deciding to not implement them in the main file system used by an operating system has the immediate consequence of disqualifying this operating system for the use on a file server (a.k.a. NAS), which is an application domain where the *BSD operating systems have been traditionally very good.
The reason is that whenever such a NAS would have Windows, Linux or FreeBSD clients, transiting any file through that NAS would potentially lose data.
In general, in the documentation of any file system, the lack of support for features like extended attributes or access-control lists must be displayed very prominently, to warn any potential users about the risk of data loss during copy operations (because the file copy commands are usually stupid enough to not inform the users whenever they are stripping file metadata, so such a loss may be discovered only when it is too late).
Anything non-native to OpenBSD can still be stored in muxfs as an archive file. You would need to ensure to pass the right arguments to the archiver to preserve the attributes in this case.
However, when instead of being compiled from sources, precompiled binaries are used, one must verify if the support for extended attributes has not been disabled, as it may happen in some misguided Linux distributions.
Besides such precompiled binaries where the xattr support had been disabled, I have also seen various GUI-based file managers that lacked support for xattr and which could strip them during copy or move operations.
The older tar and cpio archive formats do not support extended attributes, and many older tar programs support xattr, but by using tar or pax format extensions that may be incompatible with other tar implementations.
While it is important to be aware of these caveats, otherwise you may have unpleasant surprises, like I had many years ago, when I was copying files between different users via /tmp, and I could not understand where the files were losing metadata (and also their timestamps were truncated), until realizing that /tmp was on tmpfs, and copying to /tmp was silently stripping the extended attributes and truncating the timestamps (the latter might no longer be true today), with a few precautions it is possible to use extended attributes without problems on Linux and FreeBSD.
Before using extended attributes, I had also used a database file, but that had the disadvantage of being updated continuously all the time, even for file operations that did not change the file content, e.g. when renaming or moving files.
(2) muxfs uses sequence numbers to count the write operations performed on each mirror. Upon failure to mount due to the mirrors being out of sync a report is printed comparing the first mirror with the first non-matching mirror, and this includes their sequence numbers.
> muxfs needs you!
> No filesystem can be considered stable without thorough testing and muxfs is no exception.
> Even if I had tested muxfs enough to call it stable it still would not be responsible to expect you to simply take my word for it. It is for this reason that I do not intend to release a version 1.0 until there are sufficient citations that I can make to positive, third-party evaluations of muxfs.
> This is where you can help.
> I need volunteers to test muxfs, provide feedback, and periodically publish test results.
One question: do you plan to implement "concatenation" of filesystems, so you can build one very large muxfs system (10s of terabytes) ?
I don't plan to add "concatenation" as you have described, however this can, in theory, be approximated by layering muxfs on top of multiple RAID0s.
Except this is not sufficient. Flash storage for example is especially susceptible to random bitrot of data over time regardless of whether or not it is ever accessed or even powered on. Ever tried to plug in an old USB stick or SD card only to find out it was totally busted or unreadable? Scanning the entire filesystem and re-checksumming everything is therefore completely necessary.
> I decided it was finally time to build a file server to centralize my files and guard them against bit-rot. Although I would have preferred to use OpenBSD due to its straightforward configuration and sane defaults, I was surprised to find that none of the typical NAS filesystems were supported.
OpenBSD does not support ZFS.
/usr/local is not small at all by default.
On the other side, ZFS is an overly complicated behemoth, that wants direct access to the block device. Meanwhile `muxfs` works with any already existing file-system (local or remote) and just provides the checksums. So both serve different use-cases.
btrfs kinda blew up, but it would be nice to have a good and simple reliable filesystem that actually fits in with the others. ZFS is what we're stuck with till then.
But yes, bcachefs is somewhat interesting. Or maybe btrfs manages to clean up their act one day.
There were RFCs on proposals to address both the write hole and quota issues on LWN this year, with the write hole fix already having draft patches see "raid tree".
BTRFS is fine if you are not doing things that can hit those edges.
Miles better than all that geom_xxx RAIDs (I've been maintainer of geom_raid5, mind you), chipset RAIDs and FFS2 SU+J stuff, which is not completely stable even right now — I've got a ton of erros from forced foreground fsck after "normal" background fsck completion as soon as 2019 (I don't have single R/W FFS2 after that, and I'm happy!)
With all my due respect to McKusick, all this modernization of FFS2 (SU, snapshots, SU+J) always was very fragile, and software RAIDs implemented as GEOMs is much worse than ZFS VDEV layer.
Linux-induced changes downgrade ZFS performance a lot, though :-( Another level of indirection in ARC is really big deal.
What are simpler solutions to the problems that ought to be solved?
ZFS has countless bonus tunables, several types of caching distinct from the kernel VFS cache, its own write logs and special devices, multiple levels of topology (datasets in a pool consisting of vdevs consisting of drives), deduplication, compression, etc.
It is also not at all user friendly. When set up right (and no changing your mind on setup), and when fed enough resources, it does it's job well, but simple or elegant cannot describe it.
> A CoW filesystem itself is not much more complicated than a plain filesystem.
And yet there isn't one out there, there's pretty much only ZFS and BTRFS, the latter having been in a state of almost-but-not-actually-working for over a decade now.
bcachefs is the only contender and it remains a single-developer effort with little mainlining progress in the last few years.
I think the complaints levelled against ZFS is a little unfair though. I agree that there are more elegant ways to implement ZFS but actually what we have already works really damn well. And the comments about the CLI being hard to use is weird because having used a hell of a lot of different file systems over the years (including BtrFS), I’ve found ZFS to be remarkably easy.
ZFS has saved me from a number of hardware failures. If it really were as bad as the comments on here have made out, it’s have lost data several times over.
And what was would be a reasonable response to you? Pasting a novel filesystem implementation as proof that the existing ones are overcomplicated? Opinions do not have a burden of evidence.
There is a handful of CoW filesystems out there, showing that it is certainly not an insurmountable issue to write one. Rather, the problem is stopping people from doing more at this point, keeping the design simple instead.
That we don't have something better yet is likely a result of writing filesystems in general being rather laborous to do right regardless of CoW, and being incredibly unrewarding - few care about filesystems unless it's broken.
ZFS tries to solve every filesystem problem and actually doesn't even do a terrible job at it, but it can be a bit of a beast due to its high complexity and that it doesn't integrate well with the rest of the system.
Na...it's not overly complicated for what it is, but yes it is a behemoth.
>that wants direct access to the block device.
Yes for high-performance "enterprise"-setup's it is preferable, but absolutely not needed.
> Meanwhile `muxfs` works with any already existing file-system (local or remote) and just provides the checksums.
That i think is the winning point here, just add bit-rot protection to ffs.
You can set up a ZFS pool backed by files[1]. Probably not something you should do with data you really care about, but it's possible.
[1]: https://linux.die.net/man/8/zpool (Virtual Devices)