Why are there no true cross-platform filesystems?
yulian.kuncheff.com
yulian.kuncheff.com
If two operating systems need to share files, they're usually on different machines, so it's simpler to copy the files or provide a network interface to them, rather than share disk-level access.
Getting different OS makers to standardize (and therefore slow innovation) on something as performance-impacting as the file system isn't worth the benefit.
All filesystems have file size limits - it's not an artificial constraint, it's a design tradeoff in terms of the size of the metadata that the file system requires.
Ding ding ding. Much easier to have a Samba server then try to make a FS work across multiple OSes. That having been said, I haven't really had that much issue with using NTFS.
provide a network interface to them
Another thing is evolving features, Windows is particularly good at this: more and more features are added to the FS, if you don't keep mini VM and native Installation in sync, weird stuff could happen.
I use dual boot since many years, but I have stopped sharing partitions completely. If I need interoperability I use the Cloud, USB sticks and "normal VMs". It's fool-proof, safe and low maintenance.
I have generally found it safe to read from my NTFS drives in Linux (I avoid writing if at all possible). This time, however, when I went to load the result of my work back in Linux, it could not read the folder (Thunar reported some generic IO error). Going back into Windows to investigate, Window's opinion was "It smells like Linux has been here! I can't STAND that guy or anything he touches, so I burned the folder just to be safe. You can thank me later."
As best I can tell, it had something to do with a combination of:
1. Windows Update needing to do things upon shutdown and startup, but then not booting back into it for a month.
2. Linux having been in hibernation rather than halted. Might have had some access bit set on the NTFS drive when it hibernated? I don't actually have the domain knowledge to claim that is what might happen, so it's a complete hunch.
This was really just a rant to get out the anger at lost work. This article was quite timely, and while I'm not glad to see people share my pain (because it's still pain in the end), I hope we one day see improvements.
Other than that - my experience of late has been that ntfs is safe both to read and write from linux -- but as I now encrypt all my filesystems, that doesn't really help (no bitlocker support). If I actually needed to dualboot -- I suppose a separate disk/partition with truecrypt might do the trick.
My solution for the past several years have been to have all "interesting" files on a separate linux file server -- and access it with cifs/samba and/or ssh.
Not a chance. Hibernate is just like Suspend to RAM: freeze all processes but core kernel, blit RAM to disk (usually stored in swap) and send poweroff command straight to CPU. Of course it doesn't unmount anything, hence there are FS data structures live in RAM. Booting an alternate system and accessing a filesystem open on the other side is like mounting twice the same filesystem and writing to it. It will blow up on you.
"If you have network shares or external devices that should be unmounted before suspending, list them here."
"Unmounts any filesystems of the given types. This is most useful for network filesystems such as smbfs and nfs."
Windows 8 will in fact do hibernation when you think you have turned off the computer.
Bottom line: You can't dual boot safely with Windows 8.
This pretty much sums up Microsoft's reaction to Linux in a nutshell. It's precisely why articles like the one we are commenting on exist. The answer to the question that the article poses is "Because Microsoft hates competition and acts like a petty spoiled brat. Get back to us when they grow up."
Although the rational here is fairly clear: having good linux support for Azure helps Microsoft sell its cloud service. Having good linux support for Microsoft filesystems helps people move away.
[1] http://www.zdnet.com/blog/open-source/top-five-linux-contrib...
[2] http://www.geekwire.com/2012/surprise-microsoft-list-top-lin...
Find a cheap/old box, put your favourite flavour of Linux on it, add a few TB hard drives, and run Samba. If performance is an issue, use a mobo with SATA 3 controllers and a few gigE ports, and add additional network daemons for more "native" networked filesytems (NFS, AFS, etc.) as necessary.
I was very intentional in suggesting it be a separate physical device.
The author of this article seems to have mistakenly dismissed their stuff as FUSE based, which it doesn't seem to be. It's also cheap.
They also have HFS for Windows, haven't tried it even though it came free with the NTFS for OSX driver.
Not associated with Paragon in any way, just a customer.
I think that's the way of the future, not cross-platform filesystems.
I have an i7 920 and 16 gigs of ram, and I have about 50% to the guest, and Visual Studio and Xcode, along with simulators is pretty messy. Running a VM within a VM can do that.
Its the only reason I do a triple boot like this. Otherwise, I would definitely go pure VMs.
Though Parallels on my laptop does a bangup job with Visual Studio.
But then again, the main problem is huge files and lots of writes.
I don't really find this true anymore. I run a retina MBP with VMware Fusion and VS2012 in Win7 (Boot Camp partition running in VMware) runs great, with very solid perf when running my OpenGL-based engine. Games run very well, too--the Steam selection on OS X is pretty poor, but I hate dual-booting, and the perf/quality tradeoff is very acceptable to me under virtualization.
There's a free implementation available for Linux [1] (actually in the default apt/yum/whatever repos for many distros) and it ships out-of-the-box on recent OS X installs [2].
0: https://en.wikipedia.org/wiki/ExFAT 1: https://code.google.com/p/exfat/ 2: http://www.macrumors.com/2010/11/11/mac-os-x-10-6-5-notes-ex...
EDIT
Yes, it is mentioned in the article. The article sucks. It does not define what "modern" features are of a filesystem, and the author clearly sets off with his end goal in mind ("there is no cross-platform FS") randomly removing FSes from the consideration for vague and hand-wavy reasons. This applies to several of the entries in his list. I mean the guy thinks UDF will do the trick, but exFAT is not good enough? Come on!
The author mentions journaling and extents (particularly important for his torrenting use-case to prevent fragmentation), which I feel are the way he defines "modern". exFAT also supports neither transparent encryption nor transparent compression.
I've had at least 3 (I'm not sure, maybe more) Seagate internal HDDs failed on me in the past several years (I don't do anything remotely crazy with my MacBook Pro or other machines). And I've had a 1TB external HDD and 3 USB disks ruined by ExFAT (yes, I always 'eject' the volume before yanking in out from USB port).
I don't know what the hell is wrong with either of them - Maybe I've been unlucky. But that's the truth, and I'm sure the problem wasn't from my side.
However the bigger question IMO is: Do we really need native windows nowadays? If we ban windows into VMs and only use OSX + linux natively, HFS is indeed a solid option. The only thing hindering us so far from using windows VM-only was gaming and audio/video rendering software. Audio is very niche and the rest should soon be solved with increasing support for PCIe passthrough.
I'm going to raise the level of meta-discussion here and say that it's even more hipster (my subjective assesment) to use that subjective assesment (hipster) of other subjective assesments (hate Windows).
One can do GPU passthrough with Xen[1] if hardware allows, but that's a lot of footwork to set up and configure. Performance near native through from reports of those I know that have done it. Ubisoft gave a demonstration a while ago of Crysis running through Xen and GPU passthrough[2].
But maybe you have found a way and you can then prove me wrong: Does iWorks work in your virtual box VM? Does XCode including iOS simulator work well? Is the performance acceptable? How about the iLive suit? How about unity development? How about the clipboard - does it keep its 'PDF snippet' functionality (no rasterization of fonts, original quality images)?
See, there's tons of OSX exclusive software that's quite demanding in terms of media and hardware acceleration, simply because of the OS's highly integrated character. Unless someone can show me a solid OSX-in-a-box I doubt that it's a good experience.
Yes.
> Is the performance acceptable?
It is for the above. Not so much for anything demanding.
You're kinda missing my point. If I wanted to work in OS X, I certainly would never virtualize it. If I occasionally need to check a website or see how it would look like on an iPhone, then virtualization is a good option.
There are various reasons why this doesn't exist, at least not beyond compromise solutions such as vfat. A transfer partition and use of archive formats would largely suffice.
The larger problem though is expecting to be able to manage the same storage on a multi-boot system from multiple operating systems. It's one of the reasons that multi-boot is a poor alternative to virtualized systems.
The author could configure his alternative OSes as VMs, and transfer files via virtual networking, either via transfers (scp, etc.), or via mounted networked filesystems (CIFS/sambafs, NFS). Compatibility issues are handled by the networking protocols (or by using archive formats), and simultaneous access to multiple environments is available.
Maybe somebody here has the skill and time to fix this.
Neither of those seem terribly useful for the "UDF filesystem on a shared hard-disk partition" use-case we're talking about, so I don't think Linux's limit of UDF 2.01 is very important in practice.
Now I've never gotten it to work on OSX though, which is by trying my friends machines. I've always used the linux udf utility to format it, and have tried both with and without a partition table. It seems OSX, unlike Windows and Linux, actually respects the partition id.
Rather, what file format do you use on your portable drives when you use all three operating systems on different machines in different scenarios (say Windows at work, OS X at home, Linux on servers)?
I'm sure that starting a virtual machine to share EXT4 drives over the host-only network works (if you have the option of installing VirtualBox everywhere), but I'd be weary of network file system limitations and issues and overall bad performance.
A $85 USB 2.5" drive carries a terabyte of data, fits in your pocket, needs only a single cable to use (and no external power), and transfers data faster than 100MBps ethernet. Or if you really care about speed, $40 gets you a 64GB USB3 flash drive that's faster than gigabit ethernet. The problem is there's no filesystem you can write on these drives that every machine can read.
Personally, I gave up on manual organization of my file system last year, and learned how to love built-in search. This works great, as the rough index is in your head anyways, so you can just take a single token of the filename, type it in, and open it up; massively reducing both access time, and eliminating organization entirely.
I have some parts of the same project spread out through 3 folders: not in Dropbox, in Dropbox, in Public subfolder of Dropbox. It'd be really nice to have a unified view on those things. As in apply 'Dropbox' or 'Dropbox/Public' labels to some files.
I know Dropbox has shared links, but I don't want to have to create and manage those all the time. It's just an example.
Finally, I think a great way to organize files is by time created/modified. If you exclude the system or program generated files. Quite often I just want to find a file that I know I saved a few days ago (but no idea in which folder).
Therefore what you need isn't a new file system but a file system browser that allows you to specify a (set of) file(s) using tags.
Considering how easy it would be to do just this using fuse, I'd guess some Linux folks already tried this, does anyone know about projects in this line?
https://github.com/chujoii/xattr-tag/
I suppose adding a gtk-gui on top of this would be possible.
http://web.archiveorange.com/archive/v/wNWnxu100AzH1ItS2Dk5
There have been attempts to remedy those problems by both individuals like Nelson [2] and corporations like Be, Inc. [3] and Microsoft (which may or may not be related to the fact that Nelson's book Computer Lib / Dream Machines was very influential at early Microsoft and Microsoft Press even republished it). It truly is lamentable that Microsoft didn't bring one into the mainstream with WinFS [4].
[1] http://www.theatlantic.com/magazine/archive/1945/07/as-we-ma...
[2] http://www.google.com/patents?vid=6262736, http://www.xanadu.com/
[3] http://www.nobius.org/~dbg/practical-file-system-design.pdf, Chapter 5
Q: What one Microsoft program or product that was never fully developed or released do you wish had made it to market?
Gates: We had a rich database as the client/cloud store that was part of a Windows release that was before its time. This is an idea that will remerge since your cloud store will be rich with schema rather than just a bunch of files and the client will be a partial replica of it with rich schema understanding.
-- http://www.reddit.com/r/IAmA/comments/18bhme/im_bill_gates_c...
Unfortunally, this isn't very helpful - since you need to run either BeOS or Haiku. If I'm not mistaken, there's a FUSE port of BFS to Linux as well.
There are stable implementations for Linux, Mac OS X, Solaris, and BSD (at least). You can export a filesystem from any one of these, unplug the hard drive, plug it into another computer, import the filesystem, and go. Doesn't even have to have the same CPU endianness.
Besides being designed from the ground up to be operating system and CPU endianness independent, it does a ton of other stuff that no other filesystem does. It's a total rethink of the entire filesystem / volume manager / RAID pool concept. All of these things are merged into ZFS, with a user interface that's both radically simpler and far more powerful than managing all of those things seperately.
You can easily add capacity to existing datasets (up to 256 quadrillion zettabytes, it's 128 bit!), do mirroring and striping at the filesystem level, and add hot standby drives. You can combine drives of different sizes into storage pools. You can do rolling drive size upgrades in the future to expand the filesystem size when drives become cheaper, without taking the filesystem offline. It has built-in pervasive checksumming and self-healing data corruption detection, there's a strong focus on data integrity despite flaky hardware, writes are atomic, you can take snapshots and roll back to previous filesystem states, there's an advanced adaptive memory caching system that's far beyond simple least-recently-used caches..
Is that not enough? How about built-in encryption, data compression with your choice of algorithms, deduplication of file data... You can add a small SSD drive to your huge traditional hard drive based dataset to act as an L2 cache for fast, durable writes, fully online maintenance and upgrades, builtin SMB and NFS sharing, you can stream updates...
How does that fit the 'One file system on Linux, OS X and _Windows_' requirement? ZFS is useless for this scenario. No need to sell it if it cannot even start in this competition.
Edit: On top of that, some features that you're selling aren't completely available cross-platform (i.e. builtin NTS/SMB support on Linux, encryption is(?) Solaris only iirc).
They can be quite fast if you aren't running them over a physical network, such as between host and VM operating system on a single physical machine.
My preferred solution to your use case would be to run Linux in a virtual machine, configure the VM platform separately in each host OS to give the guest direct access to a dedicated raw partition in ext4 or any other Linux-compatible format, then serve Samba or some network filesystem to the host from the guest. (Of course, if Linux is your host, you can just mount the dedicated partition directly from the command line or fstab.)
Back when Windows was my main OS, I liked Colinux [1], a port of user-mode Linux [2] which allows you to run the Linux kernel as a native Windows binary.
But nowadays my first recommendation would be Virtualbox. You could also try qemu, it might have better Mac support.
Linux ZFS is still quite young, only supports a very early version of the fs, and can't be updated to the newest without serious reverse-engineering.
As for ZFS on windows, I definitely would not hold my breath.
I ran into the same frustration as the author and gave up. My notebook triple-boots Linux, Solaris 11, and OS X (I have no use for windows) and would be beyond thrilled to have ZFS shared among them.
One other problem with the idea though... you'd have to export the shared fs each time you shut down and import it into the next OS you booted, which would be kind of a pain (of course you could force the imports, but that's ugly and it only saves one step).
Which implementation are you talking about? See https://launchpad.net/~zfs-native/+archive/stable . Zpool version 28 is old now? The only implementation of 31-33 is in Solaris! See http://en.wikipedia.org/wiki/ZFS#Comparisons
And yes, I am using Solaris for comparison... I mentioned that I'm triple-booting with Solaris in the post you replied to.
I'm glad so see the development has caught up, but it is still quite young.
That joins to the kernel's world at the hip.
Personally, I'm glad to have left the time of dual booting operating systems behind me - it's a complete PITA. A Linux box with a bunch of disks under samba/ext4/lvm/mdadm is much more practical IMHO.
There are at least two filesystems, intended for situations where several computers share block-level access to the same disks over Fibre Channel or iSCSI, that are truly cross-platform, allowing volumes to be used, simultaneously if desired, from computers with different operating systems.
Quantum's StorNext File System is available for Linux, for Microsoft Windows, and for several UNIX systems, including OS X, where it's bundled with the OS and known as Xsan. SGI's CXFS ("clustered XFS") is similarly compatible.
Both of these filesystems are modern, in the author's sense that the filesystem does not impose a practical limit on file or volume sizes.
To those wondering why VMs are insufficient, if you have work (or gaming) to be done that requires maximum performance in multiple operating systems, a VM is not fast enough. I develop software for my startup in Linux, write client software in XP, and do video work and game in all three.
We will obviously run into issues where different OSes have different concepts of what the filesystem needs to do (such as differing security models), but at the very least we should be able to get the common interfaces standardized.
Use an API for cross platform sharing layered on top of a filesystem: Either a Virtual Machine sharing protocol (aka 9p virtio) or a real network protocol AFP/SMB/NFS. Anything else is taking your data in your own hands and is antithetical to the design of modern OSes.
I find that I only lose about 5% efficiency compared to booting up Linux and copying without overhead.
This is probably the reason only FAT fits the bill: it's a lowest common denominator that has so few features that it can appear "native" on all platforms.
There's a pipe dream. It'd be great but I can't possibly imagine it happening. I have to think that any spare FS effort is being spent on ZFS (OS X) or ReFS (Win8/Server2012). I'm still rocking out with a 12TB RAID10 array and another 6TB RAID10 array that I gave to my parents. Haven't had any issues yet.