Show HN: Reimagine the File System | Hacker News Reader

Show HN: Reimagine the File System

amoffat.github.io

49 pointsdaenz13 comments

Neat. As a remark, this is how LDAP is structured: entries (files) are "tagged" by attributes (directories) and you access them with the same pseudo-paths which are actually conjunctions of filters.

I must say honestly i'm not so fond of using ordered sequences (posix paths) in place of the unordered sets which they should be. I guess my question is really why you wanted to make a fuse fs out of it (versus say some tag-based filtering script like notmuch). (just for fun is surely a valid answer)

Because by making it a regular fs you immediately get support in every piece of software.

Every file chooser on your system suddenly knows about tags and gives you a way to query them.

Basically what Spivak said. I was interested in adapting an existing interface (a filesystem) to do the set intersections because it means more software could benefit from it without additional work.

This is basically to files, what gmail has done to emails. (If you can remember emails before the gmail era)

Opera Mail did that too, and before GMail, I think. A single email database with tag-based filtered views on the UI end.

Have you performance benchmarked this? Would be interested in giving it a try if performance is not compromised. Currently use XFS.

I can’t speak to the implementation but in principle, a filesystem essentially being a specialized database, there’s no reason techniques commonly used for SQL optimization couldn’t be applied here. It’s fundamentally set theory, where you’re obtaining your results by producing subsets rather than traversing a hierarchical tree of branches and end nodes. Of course that implies some pathological conditions (some superset/subset scenarios will produce too many results to be useful or to optimize resolution) which would produce poor UX, but sane defaults and heuristics could alleviate that.

Again not speaking to the project itself, but in principle achieving good performance on something like this would boil down to “do what SQL databases do, then design efficient schemas and queries for the use case”.

Not formally, no, but anecdotally, I do notice some performance impact with with many files and tags, and some of the more aggressive file browsers like to depth-first cache directories, which can be involved when directories are generated dynamically and extremely deep. I am actively looking for solutions to that!

Fortunately you can try Supertag out without touching your original files, as Supertag will use symlinks to your original files, so you don't need to move them.

You should only use raw block number instead of filesystems this way you would get even better performance /S

The Disco MapReduce framework has a tag based distributed file system.

https://github.com/discoproject/discodb

I don't know enough about filesystems to speak as to whether it's a good or bad design, but I quite like the concept.

Cool work!

It’s a great design for certain use cases and fundamentally incompatible with/needs adaptation for others.

It’ll work well for:

- Aggregate workflows (i.e. composing heterogeneous collections)

- Library curation (i.e. individual or mass categorization)

- Content discovery/rediscovery (i.e. find things based on metadata rather than location)

It’ll either poorly serve or need additional consideration for:

- Contained dependencies (i.e. any kind of encapsulated package that has internal references)

- Structural authorization (i.e. a given user may or may not be permitted access to some or all functions based on the branch/leaf they’re trying to access)

- Structural layout (i.e. the user has a hierarchy in mind which they wish to reify)

I’ll leave the above comment intact from here but I already edited once and still regret the word choice “fundamentally incompatible”. As with any data structure, you can make adaptations and accommodations to support use cases that seem to conflict. There’s always a memory/performance/impedance mismatch/experience cost for any abstraction, but there’s no reason a set/tree hybrid couldn’t provide for both families of use cases, given those costs.

Reply on news.ycombinator.com