Set Based Filesystem idea
aleveo.com
aleveo.com
"File indexing and searching for Plan 9", http://lsub.org/ls/export/tags.pdf describes mostly text indexing, but the interface to searching is through a file system (unfortunately not described in much detail at first glance).
only now do i remember that wrote something related once: attfs, http://www.ueber.net/code/r/attrfs. that's for inferno, using the styx/9p protocol. i never polished it, am not currently using it, and the interface the file system provides is not ready for casual file browsing, though that's mostly a detail (from an amount of code point of view). the things stored are strings though, not files. but the illusion that it stores files is easy to build.
I've only ever used it with wmii, but I've been looking for an excuse to write a Lua wrapper for it. :)
I think somebody was working on one in lua, but I don't know if they finished it.
If you write a Lua implementation or a lua wrapper for an existing C implementation let me know.
What do you think? In the mean time we can build some prototypes on different platforms.
The real reason why we don't know how and if this interaction model would work in practice - is the obvious absence of implementations.
A Lua client library for libixp should be pretty straightforward, looking at the source for ixpc. I might have that together this afternoon (particularly if my fiancee continues napping ;) ).
As for a server library, I'm still thinking about it, but passing an init function a table with callbacks for the various messages would be pretty straightforward - similar to LuaExpat's design (http://www.keplerproject.org/luaexpat/manual.html).
I'm silentbicycle on twitter, FWIW.
The walk function (which returns an iterator, successively returning tables with each file's stats in a directory) is not complete, it's terminating after the first file sometimes. I'll smooth that out once the server side of the library is together and testing can be self-contained.
I'd like to work on the server library sometime next week, but I'm pretty busy. Is there a better place to contact you with updates?
"ConcepOntoFs: A Semantic File System for Inferno", http://lsub.org/iwp9/cready/abheyshahplan9.pdf
this paper also has references to other papers on the topic.
This is why I over stress the need for use cases.
The reasoning behind this idea is how I use my files, the intuitive processes to organize and later find, the flow of thought and command.
Of course, a relational FS would still be broken up into namespaces (tables!), much like conventional filesystems frequently occupy multiple partitions. /sometable/tag+tag2+tag3 , for example, so searching for a file wouldn't necessarily mean searching every drive. There's no reason that making lots of small tables couldn't be cheap (they don't have to be disk partitions), and that would help constrain searching quite a bit.
Also, research into journaling filesystems has major parallels to database transactions.
I see someone hasn't read a single thing that Ted Nelson has written just yet.
Indeed I haven't. I have to change that, thanks for the tip.
Wouldn't it be possible to implement the filesystem with tags and then specify an ordering of the tags to map the graph into the usual hierarchy? The first part of the URI would specify the mapping (maybe just a name for a particular mapping), and the second would traverse the hierarchical view in the usual way.
This way, legacy apps could still think in terms of a tree while exposing a more powerful structure to new apps.
This was my biggest concern, to be backward compatible with legacy apps. And it is possible.
"C:\Program Files\X\Y\Z" - "C"+"Program Files"+"X"+"Y"+"Z" "/usr/local/files/" - "usr"+"local"+"files"
The only thing is that "usr" owns the file and so do "local" and "files" but the app doesn't care about that.
Legacy apps do not access the files directly but through the filesystem. If the filesystem knows how to convert these conventional paths into sets, there is no problem.
-- Ron Minnich
What I want to say is that Documents -> University or University -> Documents is the same, thus it makes sense. In real life the order doesn't matter.
Thus it opens a path for better HCI: computer, give me all documents from the university (all university documents)..
I am curios in what perspective you see this failing?
You might be interested in the filesystem research associated with plan9. :)
src/share/test.cpp share/src/test.cpp
Also certain implicit build systems, rely on the hierarchy to know what is parent, and what is child. You lose that with sets.
I still think it's cool idea, but not for all of your documents...
Besides how would you deal with sharing, inheriting security attributes, etc?
If I have
share/test/something.cpp and share is shared to everyone, would
test/share/something-else.cpp got shared too the same way as the first one?
but that form of structuring has been imposed by the nature of directories/files. I am not sure what the right solution would be for that case, perhaps internal tags for a set, but that would make things complicated.
In the nature of sets, maybe this could be done: "Source Code", "Shared With XYZ", "Application1"
Sharing could be defined on the set, "Shared With XYZ" is a tag that when defined could have the permissions etc. When a file is "placed in it", it is automatically shared.
We need to get detached from hierarchical thinking to see the possibilities, which is difficult as it is all present.
Personal systems have more files on them than ever before, and fitting them to hierarchies that were invented decades ago can be tricky. These issues have been abated for now with things like OS X's Spotlight and Win7's Libraries, but my suspicion is that set-based filesystems would be far simpler than the current paradigm.
A good example off the top of my head is resolving what command-line executables are available. In most operating systems, this is done by setting a PATH variable with a list of directories that contain the executables. While this is trivial for the type of people on HN, it is far from for your average user. Instead, command-line executables could be given a specific tag, and the command-line would be able to execute any application that has that tag.
Now, whether set-based filesystems could replace hierarchical ones is a different question. Such a filesystem would diverge so greatly from current assumptions that people might be too aversive to change. It wouldn't be the first time; I think Plan 9 was light years ahead in so many respects, but it never took off because it was such a radical departure.
I started playing with tags on Aleveo for that reason, maybe I should take it further on a separate test project. It makes a big difference using it and thinking how it would be used.
Also, I am not sure what would be best for executables but how are those currently seen - as having an "+x". Having an executable tag is not much different. That also means, tags are a bit "smarter" than the regular ones seen on the web.
I am mostly thrilled by the ability to see the relations of files and folders to others. Those are not folders anymore as "storage units" but "meanings".
Also currently a file can exists only in one place. Having symlinks is cruft in my opinion, compensating for the need of multiple context presence of data. Symlink is a separate file, a pointer different than the original. Windows is excluded completely :) I think file can be in several places depending on the need. After all this is not the physical world.
FWIW, plan9 allows you to control which portions of the filesystem are mounted into the visible namespace of a specific process.