Hacker News Reader

Comment by alfredxing | Hacker News Reader

What's interesting to me is that there doesn't seem to be a modern filesystem that works across all major platforms. btrfs has Linux and this (unofficial) Windows driver; ZFS has Linux and OS X support but nothing for Windows. NTFS is pretty much Windows and Linux only for good write access, and APFS is macOS-only.

I realize that designing a filesystem and writing drivers for it are very difficult tasks, but surely it would be extremely beneficial to have at least 1 open source filesystem that has good support across all platforms?

View on HN

> surely it would be extremely beneficial to have at least 1 open source filesystem that has good support across all platforms

The comedy of this is that there are actually plenty of FSes common to all three of these OSes, they're just not considered "modern"; FAT32 is still the most commonly used FS for interop, even as it shows its age with poor support for very large files (2+GB) and file systems, but you also have ISO(9660) and (the best answer we currently have) UDF supported by all three OSes natively and openly without patents or other licensing restrictions. The latter two are not commonly used as read/write filesystems, but have zero inherent restrictions on being able to be used in such a way.

However, the problem has been largely supplanted by having a fast and well-functioning network and people moving to "The Cloud" for, well, everything. It's hard to know how much of the Great Cloud Migration is caused by these kinds of intentionally engineered interoperability fails, especially the monumentally stupid exFAT patent... It's interesting to ponder given the people likely to complain the most about filesystems today are people who want to move large files (4+GB) between OSes without anguish, where the network is still just too slow to handle the task well.

> UDF supported by all three OSes natively and openly

Wasn't there a major problem in that all the operating systems supported different versions of the UDF standard? And most of them treat it as an optical disk format, so I wouldn't be surprised if some of them acted like it was read-only.

How well does UDF perform for random-access read-write devices? It was my impression that it is optimized for read-often, write-rarely, possibly without possibility of overwriting.

ISO9660 is many things.

However, it is not, not even close to, a decent file system. (And that's before we even get into RRIP and Joliet. /shudder)

And IIRC, ISO9660 has issues with being written - the path table needs to be ordered. Yes, you can make it modifiable on the fly, but the pain to do so is just not worth it.

ISO 9660 is not at all a general purpose filesystem. Although fairly trivial to reason about and implement, little and big endian systems required entirely separate structures written to disc or files would not be accessible.

Myth 2 was a popular testing media in those days due to Apple and PC endian differences. As I recall Bungie did some clever crafting on the structures to share some of the media between platforms while keeping binaries separated. Even had a fun picture of soulblighter in Finder through icon trickery.

-wrote commercial CD mastering and storage software many many moons ago

> It's hard to know how much of the Great Cloud Migration is caused by these kinds of intentionally engineered interoperability fails

+1. The great benefit of the "cloud" and the internet in general is as an escape from vertically integrated silos. Which is why all the big players are frantically trying to rebuild some kind of lock-in along different lines.

I believe that part of the issue is that the file systems are tailored to the needs of the operating system they run on. Take file permissions. As the basic permissions model, Linux uses user/group/everyone, while Windows uses ACLs. Linux also supports ACLs but they don't work in the same way that Windows' does. NTFS is built to meet the needs of Windows, while Ext4 is built to meet the needs of Linux. You can use each on the other OS with the right drivers, but just as data drives with very roughly mapped permissions.

Hmm, that's a good point! However I don't think this would be a major hurdle: file permissions offer OS-level security only, not disk-level security/encryption, so it would still make sense for different OS drivers to implement only their specific permissions attributes.

To make your point more clear: NTFS is built for Windows, while every other modern filesystem is made for OS's with Unix-style permissions (including BSDs like OS X).

Note that Windows ACLs are much more powerful than the Unix user/group/world style permissions.

They are, sort of. Linux (at least) can do multiple ACLs per FS object (U/G/W: rwx) whereas NTFS does have a slightly richer set of possible permissions listed when you use the GUI but they really boil down to the same in practice.

Novell's NWFS and NSS filesystems have always supported trustee assignments that flow recursively at the point of access rather than the point of administration. Unless you have used either of those as an administrator involving say 1000s of people and groups and millions of files then you will not appreciate this distinction.

On both Windows and Linux, if you have to make changes to FS perms, then the ACLs have to be made to each object - file or folder. On NWFS and NSS, you only do it at a point (say a directory) and then it will recurse automatically unless blocked by an IRF (Inheritable Rights Filter - bloody stupid but there if you really need it)

The end result is that making a change to a tree of files on any Unix or Windows FS takes from seconds to hours. On NWFS or NSS it generally takes seconds (for the screen to refresh).

I have never quite understood why Linux or Windows admins (I'm both) have put up with the rubbish ACLs and implementations of "modern" filesystems. Oooh RAID in software and snapshots - oh how nice.

The POSIX ACLs thing is genuinely shit, very outdated and absolute rubbish. This is the 21st century FFS. Why on earth should you wait as each file in a collection that you have deemed as belonging to sales but be readable by fred be stamped as such? Why should users be able to see the path down to a point where they have access? Why on earth should 21st C admins have to watch a change of security requirements take from a few seconds to hours/days?

Modern FS's are so NOT 1990s and what a shame.

>On both Windows and Linux, if you have to make changes to FS perms, then the ACLs have to be made to each object - file or folder. On NWFS and NSS, you only do it at a point (say a directory) and then it will recurse automatically unless blocked by an IRF (Inheritable Rights Filter

NTFS has perms inheritance (and overriding) since forever, did you know that?

>The end result is that making a change to a tree of files on any Unix or Windows FS takes from seconds to hours. On NWFS or NSS it generally takes seconds (for the screen to refresh).

Painless administration requires careful planning, no matter what OS you're using.

I think the point was that NTFS has to actually touch the files when they inherit permissions, whereas its better for admin if inheritance works without the files needing to be touched

The inheritable ACLs have to propagate (they can actually get out of sync). This is a design choice; either you check the inherited permissions at file opening or while changing the permissions or creating the file.

Chances are you will open the file more often than you will change ACLs so they chose the latter which makes sense.

Interesting. Do you have any links to read about such implementation details?

https://support.microsoft.com/en-us/help/320246/inherited-pe...

This article is about the problems it causes (note that in Vista they changed the design so when moving a subtree the system automatically propagates permissions, removing a major cause of desyncs)

No, ReFS, a modern file system, does not use Unix-style permissions but ACL like NTFS.

And those Windows ACLs form the foundation for higher level Windows security features like Active Directory, which itself is used as a foundation for many other aspects of Windows, so those ACLs are well and truly baked into the OS.

Actually it goes a bit further than that.

Every object in the OS that can be represented by an OS handle, has security permission attached to it.

Files, registry, sockets, drivers,.....

Linux has had ACLs in ext4 for a decade now.

  setfacl g:admins:rw g:staff:r somefile

  getfacl somefile

While you may be correct, those are basic functions that do not compare to what NT has. On Linux, you have to mount the FS with ACL support (I don't know if this is the default now) and manage them in parallel to the file system permissions, complicating the security model (without even going into SELinux and SUID 0 stuff). Due to legacy reasons, Linux's security model has evolved, while NT has had the luxury of a do-over during its design phase.

My post is correct but you're right. ACLs still aren't default and indedd stuff might break because of that lack of testing.

There's also no cross-platform filesystem with strong checksums (128-bit or 256-bit) for user data, and many popular user space databases (SQLite, PostgreSQL) also lack strong checksums for user data when last I checked.

UDF https://en.m.wikipedia.org/wiki/Universal_Disk_Format

This isn't really what you're looking for, but SMB is supported on all major desktop platforms. It's also supported on all major mobile platforms through third-party apps.

Instead of moving storage devices around, you might consider setting up a NAS and transferring files over the network. It doesn't work for every use-case, but it might be a viable alternative to many use-cases.

It gets even worse if you want to encrypt the data and have a modern filesystem, with encryption that both Windows and Linux support.

Windows filesystem development is focused on ReFS.

There's a Linux driver for it, but it's proprietary: https://www.paragon-software.com/home/refs-linux/

Fat has good support across all platforms.

exFat works reliably across all three.

Sadly, exFat doesn't work reliably on Windows as I've had random software fail when data is stored on a drive formatted with exFat.

Mostly that's the software's fault, mistakenly assuming anything "not NTFS" is FAT32 and insisting I need to "upgrade" my drives, even though I'm only using it as project storage.

I honestly don't know why user mode software can tell what format the volume is in, and wouldn't just expect the OS to handle it or provide various feature flags like VOLUME_SUPPORT_LARGE_FILES or so.

'Random software' should not know anything about which file system your drive has. If it does (..say.. some kind of whole disk backup utility), and you are using an unexpected file system that is not really a problem in exFat.

Just a simple counterexample, SQLite is not supported on NFS. Applications are not as insulated from the underlying storage as you suggest.

The details can easily leak, for instance with FAT in the limited resolution for timestamps or the case (in)sensitivity.

You can't fix exfat errors on linux (fsck just reports the errors).

FAT/exFAT have good support across the three OS's, but they are very far from being "modern" filesystems. One of the most fundamental cons for me is the lack of journaling on these filesystems; it's very easy to corrupt a drive formatted as exFAT, especially if it's a portable device (which is probably one of the most important uses of a cross-platform filesystem)

exFAT is patented.

Reply on news.ycombinator.com