ylow

Doing some quick googling, I see https://lwn.net/Articles/353831/ which sounds like an xattr protocol extension to nfsv3. I do not know the extent to which it is supported though...

Exactly. You build the server-side (as a FUSE alternative), and just use the OS's NFS client to mount it.

This NFS Server on one side, and the OS's builtin NFS client on the other.

I have not looked at WebDAV closely. I was looking for something that is already supported in all the major operating systems without additional libraries. I don't believe WebDAV is commonly supported out of the box?

I tested by uh... building a "mirrorfs" and compiling this project in it :-) (cargo does stress it pretty hard). I have not looked for POSIX compliance testers. Is there one you can point me to?

NFSv4 did throw out a lot of the legacy stuff, but also added a bunch of other state information like locks and delegation which is quite a bit more annoying to implement. Technically cos this is meant for "localhost-mounting-only", delegation should be easy to build (since we are expecting only exactly 1 mount). But v3 had far fewer APIs to implement, and so is a good starting point.

NFSv4 is quite a bit more complicated and is also stateful which makes it a bit more challenging to implement. NFSv3 is a good starting point, and is still supported on all the major platforms. Certainly, extending with a v4 implementation would be great future work.

Hi! Author here. Happy to answer any questions!

A PhD is one of the hardest, yet most rewarding time of my life. Its perhaps the only period where I have the opportunity to do nothing else, but "think"; where I can spend many months just reading papers, learning and attacking what seems like one little problem. It is very hard work, no doubt. Going back in time, I would still do it again. And in fact, I have seriously considered doing another PhD in an entirely different field.

But still I generally do not recommend people do it. You have to be in it, because you are very interested in the field. You have to feel rewarded by learning, and by solving problems. And also there is a lot a lot of luck involved: advisor, topic, ideas etc.

Downloading large datasets can be a pain, especially when you only need a fraction of the data. So we built git xet mount, a file system mount solution using Rust and NFSv3.

Using NFSv3 means that there is no FUSE driver needed, and works out of the box on Linux, Mac and Windows (Pro required)!

Read more about how it works at https://about.xethub.com/blog/mount-part-1 .

Give it a try yourself! Get the client at https://xethub.com/assets/docs/getting-started/installation and run "git xet mount https://xethub.com/XetHub/Flickr30k.git" to immediately explore a 4.2GB collection of images!

Even better :-) We have a paper. Its on the way.

CEO/Cofounder here. We are file format agnostic and will happily take everything. Not too familiar with the needs around pbix, but please do try it out and let us know what you think!

We are significantly faster? :-) Also, block-level dedupe, scalability, perf, visualization, mounting, etc.

Exactly for the giant repo use case, we have a mount feature that will let you get a filesystem mount of any repo at any commit very very quickly.

We are file format agnostic and you should be able to put anything in the repo. We have special support for CSV files for visualizations. Sorry for the UI perf... there are a lot of optimizations we need to work on.

Entirely in house. In Rust!

The simplest really is to chunk row-wise so adding columns will unfortunately rewrite all the chunks. If you have a parquet file, adding columns will be cheap.

Nope. No kernel driver needed :-) We wrote an localhost NFS server.

CEO/Cofounder here. Thanks! Agreed, we think data versioning is an important problem and we are at related, but opposite parts of the space. (BTW we really wanted gitfordata.com. Or perhaps we can split the domain? OLTP goes here, Unstructured data goes there :-) Shall we chat? )

By the way, our mount mechanism has one very interesting novelty. It does not depend on a FUSE driver on Mac :-)

We have found pointer files to be surprisingly efficient as long as you don't have to actually materialize those files. (Git's internals actually very well done). Our mount mechanism does avoid materializing pointer files which makes it pretty fast even for repos with very large number of files.

Yeah... The compression does defeat the chunking (your mileage may vary. We do a small amount of dedupe in some experiments but never quite investigated it in detail.). That said, we have experimental preprocessors / chunkers that are file-type specific that we could potentially do something about tar.gz. Not something we have explored much yet.

Today we just have a variation of FastCDC in production, but we have alternate experimental chunkers for some file formats (ex: a heuristic chunker for CSV files that will enable almost free subsampling). Hope to have them enter production in the next 6 months.

CEO/Cofounder here! Content defined chunking. Specifically a variation of FastCDC. We have a paper coming out soon with a lot more technical details.

Indeed, there is a lot of pain if you actually try to store large binary data in git. But we managed to make that work! So a question worth asking is how might things change IF you can store large binary data in git??

Cofounder/CEO here! I think it less about "versioning", but the ability to modify with confidence knowing that you can go back in time anytime. (Minor clarification: we are not quite storing diffs; holding snapshots just like Git + a bunch of data dedupe)