Show HN: A versioned filesystem inspired by Git | Hacker News Reader

Show HN: A versioned filesystem inspired by Git

github.com

115 pointsartagnon46 comments

OK, first off -- awesome! Secondly, I have a number of questions that probably have to do with the fact that I'm unfamiliar with a lot of this stuff.

1. Would it be possible to build this fully on top of git (perhaps using libgit2)? I ask, because the "holy" grail for me is finding some magic way for my designers to use git without knowing it. Right now my designers (and I think most non-programmer people), love to use Dropbox. Dropbox has a number of issues as a shared project tool in my perspective (not the least of which being that you are only allowed ONE dropbox account per computer, so you start having a shared folder mess and your dropbox begins to balloon in size on every computer with that account). It appears that something like this could look like Dropbox to everyone disinterested in vcs and act like Git to everyone else who cares.

2. The other reason for libgit2 is to ideally also git-push after every edit/save, to something like github.

3. Max OS X support please :)

Check out Pixelapse, which is kind of like github for designers: http://www.pixelapse.com/

Yeah I asked for an invite ages ago, haven't heard back. if they ever let me in it might do the trick.

Separately, actual github for designers would be cooler because I could:

1. Have an "artwork" git repo which my designers interact with in this way, but which I include as a submodule to a larger git repo for my programmers.

2. Use the same bug tracking for art as I do programming

3. Backup the git repo because its an open protocol vs. some proprietary thing.

Again though, I'm lazy and don't have time to write project management tools for myself so if these guys ever got around to sending me an invite I'd probably just settle.

The windows github client is relatively idiot proof. I don't know if they offer something similar for Macs.

They do, it's much nicer in fact. http://mac.github.com/

The Github app for Macs is just too bare in my opinion.

With that being said, I use SourceTree[1] every day and love it.

[1]http://www.sourcetreeapp.com/

1. I only use (a modified version of) packfile/ packfile idx from Git. No commits are actually created. It sounds like you're looking for something like flashbake (http://bitbucketlabs.net/flashbake/).

2. phoenixfs doesn't create commits, so no git-push.

3. I currently don't have access to an OS X box, sorry :(

Have you tried SparkleShare? It sounds exactly like what you want. It connects to a normal Git server and presents the repository as a Dropbox-esque folder. It has support for Linux, Windows, and Mac, although it's mainly a Linux project.

More info:

"SparkleShare is a 100% free and open source, cross-platform file sharing system started by Hylke Bons. The idea for the project came from the 2010 GNOME UX Hackfest. It's very similar to Dropbox or Box.net, but it is backed by the free and open source version control system git.

Basically, the way it works is that you will have a special folder on your computer, called 'Sparkleshare', and everything in that folder is also present in a version-controlled remote git repository. If you create a new file in a Sparkleshare folder, it will immediately be committed and pushed into the remote git repository, and everyone else who is using Sparkleshare to connect to that folder will be able to see the file you just created appear in their Sparkleshare folders. If someone else opens the file you just created and modifies it, their Sparkleshare client will automatically commit and push their changes to the repository, and you'll be able to see them in your Sparkleshare folder (and you'll get a handy notification!)

Sparkleshare is a great, free & open source way to collaborate on files, without worrying about manual version control or having to manually upload and broadcast links to your work."

http://sparkleshare.org/

Wow I will definitely take a look at this.

Congrats on a tremendous technical feat!

Any advantage of using a full-scale FS over something like git-annex (http://git-annex.branchable.com/)?

The seamless workflow that only a full-blown filesystem can offer- new revisions are created automatically when files are edited and saved.

I like it. It reminds me of VMS's versioning.

I implemented something like VMS style versioning with Fuse and Perl for a talk/demo a couple of years ago - it simply created a new version on open() if any writeable flags were passed - not robust, but functional enough to demo.

edit Possessed by VMS, not multiple VMSes.

Does your file system create revisions of the entire tree, or only revisions of individual files within the tree as VMS and RCS do?† Because there are some problems with the latter approach. One problem described by Geoff Collyer (the C News guy) is that "version numbers of related files are not kept in synch, so you can't just cp *';6' /tmp/compile".††

† I tried answering that question myself, but your program kept crashing.

†† http://9fans.net/archive/2001/11/759

Revisions of individual files only. Yeah, you can't do that kind of sorcery with phoenixfs.

† Fixed. Sorry about that.

You should be able to achieve this with Filesystem events (inotify/kqueue/filesystemwatcher). Just another way of doing things.

I'm not the biggest fan of FUSE filesystems(until highly-tuned, ala NTFS-3G), but they certainly do work.

No, you cannot achieve atomicity using these calls. By the time you receive a notification the file might have been edited twice.

If the file is changing that fast, do we really care? I'd be happy to have all my files tracked at only 5s temporal resolution...

Sure, but which 5s? :)

This is true, but not without drawbacks:

* Using FUSE you gain the atomicity through synchronicity; when a write occurs, you stand between the disk and request. In so doing, you introduce the potential for latency (nevermind the latency inherent to FUSE).

* Using fsevents, you get notifications asynchronously, so there's no I/O performance hit. You may not get the events right away, but with fsevent APIs there's a bounded queue so you generally don't miss events, unless there are too many and the buffer is overflowed[1,2]. The events don't contain the actual file data, so you will lose intermediate edit data.

Thinking about this, it would be cool to exploit a Copy-on-Write filesystem (like ZFS) to scalably add the edit data to fsevent apis. I suppose you could preserve write buffers in memory temporarily instead of CoW, but that would have a nasty memory footprint.

[1]http://msdn.microsoft.com/en-us/library/system.io.filesystem... - Internal buffer size

[2]http://docs.oracle.com/javase/7/docs/api/java/nio/file/Watch... (see the watchkey queue and overflow)

Can this handle large binary files? That's been my main problem with storing backups of my servers in git, and it's one of the strong-ish points of git-annex.

There seems to be nothing inherently limiting in Git's repository design. Interested parties can look into bup, for example, which stores large binary files in git's packfile format. https://github.com/apenwarr/bup

Yes; I don't see why not. I haven't stress-tested this enough to tell for certain though.

This is relevant:

http://www.hpl.hp.com/personal/alistair_veitch/papers/elepha...

If only half the stuff done by the likes of IBM and HP were to make it into common usage the world would be a very different place.

Nice. Why not use libgit2 (https://github.com/libgit2/libgit2)?

It's a bit of an overkill for my purposes- I've only copied out (and modified) a little of the packfile/ packfile index generation code from git.git. Plus, dealing with the actual code is a great learning exercise :)

Nice! Reminds me a bit of a better version of VMS's versioned filesystem.

Also: I'm old enough to have used VMS's filesystem. :(

Chill out. I'm old enough to have used ITS, which introduced versioning(†).

(† Apparently: http://en.wikipedia.org/wiki/Versioning_file_system)

Agreed, VMS filesystem was the first thing that came to mind for me as well. Seems rather sad that in 2012 we don't have that as a standard feature like we did back then.

For the record, you don't have to be all that old to have used VMS's filesystem :-)

My company still uses VMS specifically for the versioned filesystem.

We just retired our old vax a couple of years ago :>

Sounds a bit like ZFS snapshots[1]. I always wished that would catch on a bit more, seems like it would be very useful.

[1]: http://en.wikipedia.org/wiki/ZFS#Snapshots_and_clones

And btrfs[1] of course.

There's also git bup[2] and git-annex[3].

[1] http://www.dedoimedo.com/computers/btrfs-snapshots.html

[2] https://github.com/apenwarr/bup

[3] http://git-annex.branchable.com/

https://github.com/artagnon/phoenixfs/blob/master/delta.c#L4...

https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L59...

Missed malloc check.

https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L82

system calls like mkdir can fail. Better to check return values of such calls.

Might want to post that to the issue tracker

Extremely interesting. I wonder: when does it "commit"/create a new rev?

When you modify a file and save it.

So on close() ?

Yes. The event is called `release`: see phoenixfs_release() in fuse.c:374.

What about when a file is fsynced?

This looks pretty good. I had a look at doing something similar (but wayyyyy smaller scale than full FS), and gave up as there was many dragons. Nice to see someone pull it off.

I'm working on a similar concept which provide copy-on-write over HTTP as well: http://emr.ge

Very cool. I would like to have some automatic revisions for example of my /etc tree.

There's a tool that will commit changes to /etc before every apt-get install.

This might be what I'm thinking of: http://joeyh.name/code/etckeeper/

This is tremendously cool!

Any plans for MacOS (or other *nix) support?

There's a FUSE for OS X, so a port should be possible (should be possible for other *nix too, as long as they have FUSE). I currently don't have access to an OS X machine, so no immediate plans.

Wow, awesome project!

Reply on news.ycombinator.com