Willscott/go-NFS: Golang NFSv3 server | Hacker News Reader

Willscott/go-NFS: Golang NFSv3 server

github.com

137 pointspjf56 comments

Wasn't expecting to see this here!

This turned out easier than i was expecting it to be. It's nice to be able to mount a VFS without needing privileges on the server side. The main intention for this code is to eventually use it to replace fuse on mac, since nfs is a valid mount time for mac clients to consume.

The fuse replacement is super interesting. So you’d basically have a user space utility that provides NFS<->fs bridging? Would you support the existing fuse API? What does NFS not allow that Fuse would, if anything?

Thanks for your contribution!

one of the major attractions of the fuse abstraction layer is that once you have fuse you can interact with more exotic file system types without having to deal with new kernel drivers each time.

Instead though, you can translate the exotic file system types to nfs, and use the existing kernel drivers that already exist for mounting an NFS mount.

Would you support the existing fuse API? What does NFS not allow that Fuse would, if anything?

Very cool! With "only" 3.2k lines of code for a NFS implementation I was indeed surprised that it was so easy. As for the privileges, how do you deal with the issue of port 111 being a privileged port?

It defaults to a high port. when mounting, you can pass options to set both NFS and MOUNT port and skip the portmapper.

> With "only" 3.2k lines of code for a NFS implementation I was indeed surprised that it was so easy.

Not criticizing the projet here, which is very cool. But don't get too hasty.

This is an _untested_ (per the docs), _unoptimized_, _read only_, _server only_, NFS v3 only, implementation.

it is read-write and shares types with the existing https://github.com/vmware/go-nfs-client for the client side implementation.

optimization and testing are ongoing :)

That's awesome – better update the README. :-)

This is great! SOOO useful to be able to do mounts w/o fuse, w/o kernel extension, w/o root. This is a HUGE UX thing, specially for macs.

Fuse is a really great idea & project, but sadly today impls require a lot of install steps in some platforms, and some have painful bugs/UX. Amazing if we can mount VFSes from Go without those hurdles.

Just to lay them out, some notable other options:

- Under Linux and FreeBSD, bazil fuse offers a nice low dependency path to supporting FUSE. It is all pure Go code.

- Under Windows, you can use WinFSP, which is a rock solid usermode filesystem driver with FUSE support. Despite its name, you can use this with Cgofuse to get FUSE on Windows without CGo.

Combining the two above gets you pretty far already (though if you want to actually avoid a CGo dependency you do need to explicitly disable it.)

I think the next level would be to have a general filesystem interface (there are several options in the ecosystem already) that can be used as a common ground between different usermode filesystem and network filesystem server implementations. Then ideally, a magic Go library can exist that gives you the best possible setup with switchable backends depending on the platform and build configuration.

Right now Go itself is possibly standardizing a read-only filesystem interface, an extension of this with write support might serve as a decent jumping point for getting toward an idealized pluggable filesystem library.

> I think the next level would be to have a general filesystem interface

That's starting to take shape: https://old.reddit.com/r/golang/comments/hv976o/qa_iofs_draf...

It won't be enough for the likes of FUSE, though. Some extensions might be enough to bridge that gap, it's too early to tell. FUSE is pretty particular about things like rename(2) preserving node identity.

Another option:

Fairmount, an app DVD ripping on MacOS, uses WebDAV: https://github.com/BoxOfSnoo/Fairmount

This is cool! At a previous company we got a lot of mileage in debugging and ad-hoc tasks by exposing various things as filesystems. We mostly did webdav, but NFS is way better from a client transparency perspective. For many folks find, ls, and grep very much beat curl and jq.

I would love to understand more about what you exposed as filesystems and why. Is that on https://skife.org/ somewhere?

A nice protocol to do this with is 9p [1]. It's much easier to implement a server for it than for NFS, either from scratch or even with a good library, and it's about just as usable on major operating systems. It has a tiny API and therefore surface area, also making it great for high security applications. It’s probably the closest thing we have to a cross-platform, network transportable FUSE.

Even though it started its life as a component of Plan9, 9p is getting through a renaissance period right now: qemu, WSL, gVisor, ChromeOS VMs (crostini)... all use 9p!

[1] - https://en.wikipedia.org/wiki/9P_(protocol)

Unfortunately all the default implementations really suck at performance (for Linux and Windows, at any rate), and it’s very noticeable under IO pressure, lots of small files, etc. They're stable and seem to be correct though, which is something.

9P is fine as kind of lowest common denominator network filesystem, as long as you don’t hit its fundamental limitations, such as not being particularly Unix-compatible, semantics wise.

Considering how many things have been implemented on top of HTTP, I find it interesting that something like WebDAV (or Solid[0], or remoteStorage[1]) hasn't nearly completely supplanted NFS/SFTP/etc.

Not saying that would be ideal from a technical perspective (WebDAV at least has some issues), just surprised the ability to access remote filesystems from the browser hasn't been a bigger driver.

For example, there are a lot of interesting apps built on top of Google Drive as the storage backend, but overall the concept doesn't seem to have gained much traction.

[0]: https://inrupt.com/solid

[1]: https://remotestorage.io/

NFS is a lot faster. A custom protocol will almost always beat a kludge on top of HTTP

Yeah, that's what I was alluding to by saying it would likely not be ideal from a technical perspective. But raw TCP with a custom protocol is faster also, and yet many (most? all?) mobile apps use HTTP+JSON, even if they don't have a web app to support.

Most internet applications use HTTP because real networking is a pain, designing a custom protocol correctly is difficult, and HTTP provides enough functionality and flexibility to munge network traffic any way you want. They use JSON because it's the least pain in the ass of all the data formats.

Custom protocol over UDP is the fastest, which NFS supports, so NFS is basically the fastest (unless you get into wonky multiplex-streamed custom apps). However, UDP apps do not often work well through firewalls.

I feel like I'm fairly familiar with the tradeoffs. I'm simply making an observation that it's interesting to me that filesystem access over HTTP isn't bigger than it is.

> filesystem access over HTTP isn't bigger than it is.

I don't think it's a trade-off thing. It's conceptual; FS over HTTP is not bigger than it is because filesystems are a much more stateful concept that often has to support coordinative transactions (like locks) whereas HTTP is a stateless protocol by design, so there's fundamentally a mismatch. That's why HTTP is better matched for an object system like S3 which is capable of being transported over stateless links and gives fewer coordination guarantees and is only eventually consistent.

You can support locks for HTTP if you need them (WebDAV does), though I would argue they aren't necessary for a large number of useful tasks.

You don't have to implement all the semantics of a filesystem in order to benefit. The main benefit comes from making a filesystem-like construct accessible in the browser. As I said before, Google Drive is a good example of this (and it's an object system, so that's an orthogonal issue), but it's a complicated protocol (as is S3) in a walled garden.

I'm wondering if we are going to see NFS-over-QUIC.

Widespread adoption of QUIC may make more firewalls become UDP-friendly.

NFS is a really slow churning wheel. I think you’ll probably first see an inefficient generic option to tunnel tcp over quic (on both ends) that someone will hack to support nfs before you see an NFSv5 with quic support.

NFS is not particularly married to UDP though, unless we're talking about old timey NFSv2. There was even a WebNFS spec back in the day designed to let Java applets mount stuff over the internet.

But NFS isn't a custom protocol for any specific app either, and apps that need good fs performance (eg databases, video editing platforms, "big data" compute platforms etc) avoid it.

Implementation-wise HTTP has the advantage that the apps can tune the client code because the protocol client implementation isn't in the kernel.

In addition HTTP security (transport & user authentication) is works and everyone understands it, whereas on NFS it's a huge mess and enemy of interoperability.

In practical popularity NFS fares extremely poorly eg on AWS (how many apps use EFS vs S3).

EFS is much more expensive than S3, so there is a strong incentive to make your app work with S3 unless you absolutely need a filesystem.

As with all things AWS, it depends.

Yes EFS is 30¢ per GB and S3 is 2.3¢.

But S3 also charges $5/million object writes (and 40¢ for reads), but EFS is "free" (factoring burst capacity, per-instance maximum throughput limits, etc).

Couple that with EFS' lifecycle migration allowing you to get storage costs as low as 2.5¢/GB for many write-once-read-(almost)never after 7 day applications, things get a lot more complicated.

Mostly target audience, I'd imagine. Anyone still using NFS or iSCSI probably has specific performance or integrity requirements that those services aren't geared for, plus multiple concurrent access that makes locking and cache coherence significant questions.

In terms of low-urgency storage though, the Dropbox/Google Drive/Syncthing model of directory replication largely has supplanted SFTP, though.

There is probably some middle group of S3/B2 type storage backing some FUSE-style integrated filesystem, but I'd suspect the same problem that makes WebDAV unpopular hampers that, namely that it almost invariably sucks horribly on Windows when it works at all.

Nfs is a protocol, and there are many other interesting uses for it, for example nfs over vsock.

Almost all WebDAV implementations are a big pile of crap, and network filesystems in general are much more tricky than people realize. If somebody invents a brand new protocol that that actually works it would probably be adopted, but it would face at least as many problems as NFS does.

I don't really understand why people want network filesystems most of the time. If you try to pretend it's like real local i/o, and it's not on a perfect network, you will have a bad time. Protocols designed specifically just to transfer a file tend to be more resilient and also not expose people to the problems of programs trying to do file i/o.

I don't think you need to go full filesystem abstraction, a la FUSE. I completely agree trying to do something like access a git repo on a FUSE FS which is actually remote is going to be painful.

But just having a standardized way to do directory listings, uploads, partial writes, auth, etc over HTTP would be useful to me.

Windows 10 has a new API just for this, the integration with OneDrive (apart from being extremely obtrusive/pressuring me into use something I don’t want) works great.

It’s basically a shell component that abstracts over file selection for open/save and an api for providing the list of files, reading or writing files/parts of files, and shims around common fs operations like copy and move.

I remember trying a few WebDAV implementations back in the day -- the performance was awful, and they were difficult to configure compared to a vanilla SSH or Apache/Nginx install.

Implementing on top of HTTP is usually to make things easier for users behind firewalls, squid proxies, and the like. In my experience, those environments (schools, offices) usually had to open port 21 and/or 22 for FTP/SFTP for a web maintainer anyway...

Sun developed WebNFS [0] specifically so Java applets (remember them?) could access remote filesystems through firewalls from the browser. It's basically the same protocol as NFSv3 with some clever workarounds so that it doesn't require the separate MOUNT protocol to set up a connection (or the portmapper). That way everything runs on a single port.

[0] https://en.wikipedia.org/wiki/WebNFS

Because WebDAV implementations aren't that good and a true pain in the butt for system admins to setup. If someone made one that was half as easy to setup as say PHP, it would be doing really well right now.

Didn't see it mentioned in the README, so I'll ask here: any particular reason to go NFSv3 vs NFSv4? I'm not familiar enough with the protocol to venture an educated guess.

V4 has many more things going on (it's stateful, has a much more complicated locking system integrated, and adds some other points of complexity that get better performance). More than I was ready to take on for a first pass :)

You had me at stateful

For what it's worth, stateful is a good thing; NFSv4 leases are what allows it to safely do more aggressive caching than NFSv3. Back when it was new, benchmarks showed a good speed increase all around...

Stateful isn't inherently good or bad. It's a tradeoff. But the more programming I do the less I think it's a tradeoff worth making.

In the context of NFS, it's a trade-off that enabled higher performance (and some new features too).

When looking for something like this a year or so ago, I found [1] which supports both NFSv3 and v4 as well as p9. It worked alright in my experience though I eventually switched to ZFS which has built in support to auto-configure NFS shares.

[1]: https://github.com/nfs-ganesha/nfs-ganesha

Possibly useful existing implementation:

https://github.com/unfs3/unfs3

This has the benefits of being tested and having a working read-write implementation (along with still being user-space).

@willscott: Do you have any experience to share running this over Wireguard or similar?

Edit: Thank you!

Most testing has been on a Lan or on the local machine for local VFS use cases.

This layer eschews responsibility for multiple concurrent clients for simplicity. No promises you'll get either decent performance or proper cache invalidation when using it that way. It tends to be conservative in not filling in all the opportunistic caches.

There's a hook the backing application can use to tune reads and write sizes that becomes much more relevant in a network case. How those should be set in practice isn't something I've spent time on.

I wonder if this is a good way to support projects like Microsoft's Git VFS without requiring specific kernel drivers. Mounting a special Git NFS drive + having all the magic hidden behind NFS seems like it could be interesting.

I've always wondered if I could take the Unix "everything is a file" approach way too far and hook it up to a web service. This looks like exactly the kind of glue to make that easy...

Here's a thing like that for youtube: https://github.com/rasguanabana/ytfs

My favorite "everything is a filesystem" hack from around 2001 was cdfs for Linux, which would expose each audio track as a .wav file you could cp, and each separate data session as a subdirectory, so you could access prior states of the CD filesystem.

Good work ! PS is NFS still a thing ? I thought Gluster(FS) or S3 sorta replaced it in most shops ? I could be mistaken of course.

I’m using nfs for Docker on Mac, as its native share system (osxfs) has awful performance. Not the best setup, but didn’t find a better one yet.

Interesting... maybe give Gluster a spin when you bored of have time.

Reply on news.ycombinator.com