WinBtrfs – A Windows driver for the next-generation Linux filesystem Btrfs
github.com
github.com
I realize that designing a filesystem and writing drivers for it are very difficult tasks, but surely it would be extremely beneficial to have at least 1 open source filesystem that has good support across all platforms?
Novell's NWFS and NSS filesystems have always supported trustee assignments that flow recursively at the point of access rather than the point of administration. Unless you have used either of those as an administrator involving say 1000s of people and groups and millions of files then you will not appreciate this distinction.
On both Windows and Linux, if you have to make changes to FS perms, then the ACLs have to be made to each object - file or folder. On NWFS and NSS, you only do it at a point (say a directory) and then it will recurse automatically unless blocked by an IRF (Inheritable Rights Filter - bloody stupid but there if you really need it)
The end result is that making a change to a tree of files on any Unix or Windows FS takes from seconds to hours. On NWFS or NSS it generally takes seconds (for the screen to refresh).
I have never quite understood why Linux or Windows admins (I'm both) have put up with the rubbish ACLs and implementations of "modern" filesystems. Oooh RAID in software and snapshots - oh how nice.
The POSIX ACLs thing is genuinely shit, very outdated and absolute rubbish. This is the 21st century FFS. Why on earth should you wait as each file in a collection that you have deemed as belonging to sales but be readable by fred be stamped as such? Why should users be able to see the path down to a point where they have access? Why on earth should 21st C admins have to watch a change of security requirements take from a few seconds to hours/days?
Modern FS's are so NOT 1990s and what a shame.
NTFS has perms inheritance (and overriding) since forever, did you know that?
>The end result is that making a change to a tree of files on any Unix or Windows FS takes from seconds to hours. On NWFS or NSS it generally takes seconds (for the screen to refresh).
Painless administration requires careful planning, no matter what OS you're using.
Chances are you will open the file more often than you will change ACLs so they chose the latter which makes sense.
This article is about the problems it causes (note that in Vista they changed the design so when moving a subtree the system automatically propagates permissions, removing a major cause of desyncs)
Every object in the OS that can be represented by an OS handle, has security permission attached to it.
Files, registry, sockets, drivers,.....
setfacl g:admins:rw g:staff:r somefile
getfacl somefileMostly that's the software's fault, mistakenly assuming anything "not NTFS" is FAT32 and insisting I need to "upgrade" my drives, even though I'm only using it as project storage.
I honestly don't know why user mode software can tell what format the volume is in, and wouldn't just expect the OS to handle it or provide various feature flags like VOLUME_SUPPORT_LARGE_FILES or so.
The comedy of this is that there are actually plenty of FSes common to all three of these OSes, they're just not considered "modern"; FAT32 is still the most commonly used FS for interop, even as it shows its age with poor support for very large files (2+GB) and file systems, but you also have ISO(9660) and (the best answer we currently have) UDF supported by all three OSes natively and openly without patents or other licensing restrictions. The latter two are not commonly used as read/write filesystems, but have zero inherent restrictions on being able to be used in such a way.
However, the problem has been largely supplanted by having a fast and well-functioning network and people moving to "The Cloud" for, well, everything. It's hard to know how much of the Great Cloud Migration is caused by these kinds of intentionally engineered interoperability fails, especially the monumentally stupid exFAT patent... It's interesting to ponder given the people likely to complain the most about filesystems today are people who want to move large files (4+GB) between OSes without anguish, where the network is still just too slow to handle the task well.
Wasn't there a major problem in that all the operating systems supported different versions of the UDF standard? And most of them treat it as an optical disk format, so I wouldn't be surprised if some of them acted like it was read-only.
However, it is not, not even close to, a decent file system. (And that's before we even get into RRIP and Joliet. /shudder)
And IIRC, ISO9660 has issues with being written - the path table needs to be ordered. Yes, you can make it modifiable on the fly, but the pain to do so is just not worth it.
Myth 2 was a popular testing media in those days due to Apple and PC endian differences. As I recall Bungie did some clever crafting on the structures to share some of the media between platforms while keeping binaries separated. Even had a fun picture of soulblighter in Finder through icon trickery.
-wrote commercial CD mastering and storage software many many moons ago
+1. The great benefit of the "cloud" and the internet in general is as an escape from vertically integrated silos. Which is why all the big players are frantically trying to rebuild some kind of lock-in along different lines.
Instead of moving storage devices around, you might consider setting up a NAS and transferring files over the network. It doesn't work for every use-case, but it might be a viable alternative to many use-cases.
There's a Linux driver for it, but it's proprietary: https://www.paragon-software.com/home/refs-linux/
Back to this project, I feel there is a strong need to have a set of GUI tools (beyond Windows Shell Extensions) to take advantage of the btrfs filesystem features (like creating snapshots, sending snapshots to backup storage, etc.). I use btrfs on Linux, and the lack of good, up-to-date GUI tools to use its feature set has been a source of frustration (yes, I can use the command line, but then that would restrict the usage only to me, leaving out the other not-so-tech-savvy people around).
How is the UDF support on macOS? I ask since I relatively recently chose UDF as the Windows/Linux common HDD file system format, but I did not consider macOS.
It's totally sane. It's "like" FAT32 except you don't have the 2G filesize limitation. NTFS might also be sane, but out of the box macOS doesn't have write support.
https://en.wikipedia.org/wiki/Universal_Disk_Format
but somehow everyone thinks that it is only fod CD/DVD's.
Maybe not a deal breaker for a lot, but for crucial data, my advice will be build a cheap NAS with Linux inside, and pass all data between OSs over network
http://www.techrepublic.com/article/pro-tip-enable-ntfs-writ...
Still, a step in the right direction, at least on my book. My shared drives are all exFAT and that seems to work fine out of the box for Linux, Win and macOS.
What's interesting to me is that there is no modern filesystem that has a useful ACL mechanism. Say you have a tree of say 100M files and subdirs: now give rights at various points - it takes ages.
Your complaint can be worked around by remounting via an intermediary system and VMs are cheap. Mine is not really a complaint - per se - but a challenge: Why on earth can NSS (off of the 90s) do rich, recursive perms based on a "trustee assignment" on the fly but NTFS and Unix's finest not?
Due to past long standing enospc challenges, more experienced users are in the (bad) habit of micro-managing Btrfs manually with filtered balance, e.g. -dusage=15 -musage=15. Really we kinda need to stop doing that to find and fix the remaining edge cases; or worst case upstream should deploy a systemd timer / cron job to do these partial balances maybe once a week.
Anyway I don't balance at all on my drives and haven't run into enospc in quite a while. So chances are you're hitting an edge case, and there's a bug that needs to be tracked down if you're already using 4.12.6+.
If I had a guess, aside from "because I wanted to see if I could", it might be because of inherent differences in the way the OSes behave with regard to filesystems[0] would make a re-implementation from spec work better (faster, more reliably or whatever), though the author goes quite a ways to point out that he's used none of the original source, as if to state it's a clean-room implementation and I'd speculate (wildly) that though there are likely more than a few differences in Linux/Windows' APIs, there is almost certainly a lot about how btrfs works that isn't different when implemented on both platforms. These APIs would affect how the code is designed, and attempting to port by converting API calls that are Linux specific and plugging in similar Windows API calls could lead to a less-ideal implementation than if one read the spec and designed it targeting the operating system's API directly[1].
And then there's my original, wrong, assumption: if this individual is the only author, and it appears they might be, it does give him/her the ability to dual-license it later; i.e. offer a commercial license which doesn't have GPL-related requirements or switch it out to a less restrictive Open Source license if the author so chooses.
[0] I'm not an expert or even a n00b in this area. I, almost completely, know nothing about filesystem API differences between Linux and Windows since I don't know either the kernel interfaces for Linux or the Windows APIs related to the filesystem at any level having never written anything targeting that technology, directly.
[1] A former project that I built a long time ago comes to mind where I tasked with porting a tool to change what it used for storing records. The technology I started with had no concept of saving array records to disk, requiring a separate write for each record. Non-ideal, but not terrible because the technology was optimised at doing many small writes. The new target could accept an array of records and write them all in one call. I did a "compatible" re-implementation, replacing calls with their equivalent in the new record store and the performance made the product unusable. It wasn't until I redid all of the record I/O with the target platform in mind that things started working properly.
The FOSS community doesn't really care that much either, mostly because Linux can deal with NTFS, and the other way around happens far less, so no real push has been made for it.
Why do you also check the artifacts in?
edit: Unsupported features: 1, journal: log-based operations, external journal 2, EA (extended attributes), ACL support
There's also less motivation for implementing this filesystem in Linux at this point. It's available only on Server and Windows 10 Workstation (at least for creating -- I believe Win10 is able to read/work with ReFS in any version). It's not feature-complete yet[0], missing some important ones like deduplication. It's quite new, so there isn't a lot of it in the wild, reducing the need to make a compatible implementation for interoperability (probably the #1 reason for making a compatible version; #2 being to have a cool new filesystem with features that don't exist in currently available offerings). And being proprietary, it will almost certainly be a very difficult thing to write -- much like NTFS is/was.
[0] Not a show stopper - NTFS evolved over time, as all filesystems seem to.
> That said, isn't it true that Windows drivers are userspace-based, anyway?
No.
There have been a few projects to try to bring FUSE-like functionality to Windows like Dokan but I believe these are still too immature.
Keybase.io and sshfs use it, currently. Personally, I love it. I'm sure you're right about performance limitations, but there are very good use cases for it, like Keybase Filesystem and sshfs, where any latency introduced by the driver subsystem is less likely to be felt due to the latency of working with files over the network.
Interestingly, a set of the insider previews of Windows broke Dokany so badly that launching the driver caused everything to hang (hard -- all but the mouse stopped responding to anything, even CTRL+ALT+DELETE). It's tricky stuff, apparently.
As others have stated, Red Hat explicitly pulled engineering resources off btrfs.
ZFS just blows btrfs out of the water, there is really no comparison. As painful as it is to say, after all these years, btrfs still really feels like a rough draft of a modern filesystem.
That's no insult to the people who've done amazing work on btrfs over the years, it's just calling a spade a spade. Something was just missing in its development; maybe it was vision, cohesion, whatever it was at an organizational level, something stopped btrfs from really coming together as a first-class enterprise-ready FS. We should recognize that and let it go, as Canonical and Red Hat have gradually been doing.
Also ties up an unreasonably large amount of memory for filesystem operations. That is probably not okay with most people using a personal computer,
This is only really true when using features like deduplication, otherwise ZFS works fine on machines with only a couple gigs of RAM.
That said, if the only issue is that ZFS's memory profile is too heavy for personal systems, that seems like something that justifies some tweaks to ZFS, possibly a "home use" operational mode, rather than a completely new FS, am I right?
I'm using ZFS without incident, but my workstation has 64G, so not really a good example...
I'm not a btrfs fan myself (quite the opposite) but the gross way in which this minor Red Hat move is being exaggerated as the end of btrfs makes me think something's up. People are willing to spread the FUD a little too easily. Fact is that btrfs is still one of the top two most advanced filesystems in the world, part of the kernel, and SUSE for one is all in on it.
[1] They have legitimate reasons: it's a big chunk of code to maintain backports for and not at all popular in RHEL deployments.
I decided to go with btrfs on my work and personal computers a couple years back and haven't really had to battle it.
Just curious of your thoughts more than anything.