We are still using it for some pretty large apps. Still have not found a good and simple alternative. I like the simplicity and performance. Scaling is a challenge though.
SMB is a nightmare to set up if your host isn’t running Windows.
sshfs is actually pretty good but it’s not exactly ubiquitous. Plus it has its own quirks and performs slower. So it really doesn’t feel like an upgrade.
Everything else I know of is either proprietary, or hard to set up. Or both.
These days everything has gone more cloud-oriented. Eg Dropbox et al. And I don’t want to sync with a cloud server just to sync between two local machines.
I looked, found the link below, but it seems to just fizzle out without info.
https://en.wikipedia.org/wiki/DCE_Distributed_File_System
Anyway, we used it extensively in the UIUC engineering workstation labs hundreds of computers, 20+ years ago, and it worked excellently. I set up a server farm 20 years ago of Sun sparcs but used NFS for such.
plusses were security (kerberos), better administrative controls and global file space.
minuses were generally poor performance, middling small file support and awful large file support. substantial administrative overhead. the wide-area performance was so bad the global namespace thing wasn't really useful.
I guess it didn't cause as many actual multi-hour outages NFS, but we used it primarily for home/working directories and left the servers alone, whereas the accepted practice at the time was to use NFS for roots and to cross mount everything so that it easily got into a 'help I've fallen and can't get up' situation.
(off topic, but great username)
[0]: https://openafs.org/
IMO IBM/Transarc died for two reasons. First, there was significant brand confusion after the release of Windows Active Directory and Windows DFS since no trademarks were obtained for DCE service names. Second, the file system couldn't be deployed without the rest of the DCE infrastructure.
There was an unofficial effort within IBM to create the Advanced Distributed File System (ADFS) which would have decoupled DFS from the DCE Cell Directory Service and Security Service as well as replaced DCE/RPC. However, the project never saw the light of day.
Samba runs fine on my FreeBSD host? All my clients are Windows though.
If I wanted to have a non-windows desktop client, I'd probably use NFS for the same share.
It's one of those tools that, unless you already know what you're doing, you can expect to sink several hours into trying to get the damn thing working correctly.
It's not the kind of thing you can throw at a junior and expect them to get working in an afternoon.
Whereas NFS and sshfs "just work". Albeit I will concede that NFSv4 was annoying to get working back when that was new too. But that's, thankfully, a distant memory.
NFS support was lacking on windows when I last tried. I used NFS (v3) a lot in the past, but unless in a highly static high trust environment, it was worse to use than SMB (for me). Especially the user-id mapping story is something I'm not sure is solved properly. That was a PITA in the homelab scale, having to set up NIS was really something I didn't like, a road warrior setup didn't work well for me, I quickly abandoned it.
Samba can be. Especially when compared with NFS
> NFS support was lacking on windows when I last tried.
If you need to connect from Windows then your options are very limited, unfortunately.
SMBv1 has a reputation for being an extremely chatty protocol. Novell ipx/spx easily dwarfed it in the early days. Microsoft now disables it by default, but some utilities (curl) do not support more advanced versions.
SMBv2 increases efficiency by bundling multiple messages into a single transmission. It is clear text only.
SMBv3 supports optional encryption.
Apple dropped the Samba project from MacOS due to gplv3, and developed their own SMB implementation that is not used elsewhere AFAIK. If you don't care for Apple's implementation, then perhaps installing Samba is a better option.
NFSv3 relies solely on uid/gid mapping by default, while NFSv4 requires idmapd to run to avoid squashing. I sometimes use both at the same time.
I'd use the Finder to browse files, and for that it is terribly slow. Also without extra config in SAMBA it does litter the whole disk with DS_Store crap. I remember it was very slow that way, but have set up extra config in SAMBA (pear extension I think). Its extreme slowness may also be related to the workarounds to avoid that crap being not fully correctly configured. Also copying is comically slow, to get 4 files totaling 50kbytes can take 20 seconds sometimes. Same from a windows laptop takes sub second time.
Overall I'm underwhelmed by MacOS/iOS, this being one minor annoyance in the list. Windows and Linux both perform well and work well out of the box with my proven simple setup. (No AD)
Samba in MacOS might speed up things, but I bought that machine to get stuff cone more effectively than from Linux, and so far it didn't prove its value. Right now I'll not bother with that, as I feel that would have even worse OS level integration. Thanks for the advice nevertheless, much appreciated.
It's very easy on illumos based systems due the integrated SMB/CIFS service.
I've always thought that NFS makes you choose between two bad alternatives with "stop the world and wait" or "fail in a way that apps are not prepared for."
I do agree that object storage is a nice option. I wonder if a FUSE-like object storage wrapper would work well here. I've seen mixed results for S3 but for local instances, it might be a different story.
This is why I say there’s mixed opinions about mounting S3 via FUSE.
This isn’t an issue with a self hosted S3 compatible storage server. But you then have potential issues using an AWS tool for non-AWS infra. There be dragons there.
And if you where to use a 3rd party S3 mounting tool, then you run into all the other read and write performance issues that they had (and why Amazon ended up writing their own tool for S3).
So it’s really not a trivial exercise to selfhost a mountable block storage server. And for something as important as data consistency, you might well be concerned enough about weird edge cases that mature technologies like SMB and NFS just feel safer.
That's the opposite of my experience. Fire it up and it just works, in less time than it would take you to configure NFS sensibly.
Samba can be set up easily enough if you know what you’re doing. But getting the AD controller part working would often throw up annoying edge case problems. Problems that I never had to deal with in NFS.
Though I will admit that NIS/YP could be a pain if you needed it to sync with NT.
Everything that was supposed to replace it is so much worse, except for supposedly not being very unsafe.
Might just be bad timing then, most of my experience with it was in that v3/v4 transition period. It was bad enough to make me swear off the whole thing.
- POSIX compliant, including dotting the i's. As opposed to, say, NFS which isn't cache coherent.
- performance and scalability. 1 TB/s+ sequential IO to a single file is what you'd expect on a large HPC system these days.
- Metadata performance has gotten a lot better over the past decade or so, beating most(all?) other parallel filesystems.
Downsides:
- Lots of pieces in a Lustre cluster (typically nodes are paired in sort-of active/active HA configs). And lots of cables, switches etc. So a fairly decent chance something breaks every now and then.
- When something breaks, Lustre is weird and different compared to many other filesystems. Tools are rudimentary and different.
To get a feel for what 'life with Lustre' could be, see e.g. various 'site reports' from workshops. E.g. for a couple somewhat recent ones: https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_rep... and https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster...