Hacker News Reader

Comment by uvdn7 | Hacker News Reader

Maybe the File System shouldn’t be hierarchical but rather relational as well.

Microsoft Sharepoint uses SQL Server for all user Item storage, so it's an example of an all relational file store. They did this around the time they were also experimenting with the Cairo a sibling comment mentions. Also as another commenter noted you query on the metadata. Folders/directories for instance are represented as metadata. Installations can also describe very detailed ontologies including ones provided by third parties. It's a lot of work.

Edit: I think I was actually thinking of WinFS which came out of Cairo later, around 2000.

That means onedrive for business uses sql server as it uses sharepoint online, interesting.

That's an interesting point. As I recall (unfortunately it's been a while). There was tension at first who would be top dog. SharePoint or OneDrive, and OneDrive won, so SharePoint Online is actually mapped over OneDrive. MS could do that because ironically even though SharePoint On-prem had been entirely SQL Server underneath, SharePoint On-prem devs generally never wrote a line of SQL. They interacted with the SharePoint Object model and that was basically mapped over OneDrive. What OneDrive is underneath I don't think MS had to be as forthcoming about because of it's cloud-based nature. I was on my way out by that point though.

Microsoft wondered that too and for a while developed "CairoOFS" as a possible replacement for NTFS. It was intended as a relational "object filesystem" https://betawiki.net/wiki/Microsoft_Cairo

And then we could have fun looking at the execution plan of our file access. :D

Meant in jest; though I think the idea of a "one true way" to access files is a pipe dream. The hierarchical works better on my computer than whatever scheme we've cooked up for our phones.

> The hierarchical works better on my computer than whatever scheme we've cooked up for our phones.

What do you mean? Hierarchical is how I access files on my phone. Is that an Apple thing?

I thought phones had moved to an "ownership" model where access to files goes through the applications. That said, trying right now, I see there is a files app that seems to be mainly types. I'm assuming that is actually by folder?

> I thought phones had moved to an "ownership" model where access to files goes through the applications.

There are many file explorers for Android, and they all have a normal browsing interface. The only "ownership" thing I’ve encountered is with the Nextcloud client, where it makes sense as those directories and files are not necessarily on the phone.

I’m only on Android 10, but I think I would have heard if Android 12 completely changed things.

Yeah. I can only think I was confused by the few times I've tried to mess with the data. I thought I read they were trying to not go folders based.

The only thing the relational models really gives you is the ability to join two relations, which lets data decompose for storage but re-compose for many different uses. I'm not immediately seeing how that would be useful in a file context, where generally you want to look up a specific blob of data (aka a file).

A tree-based file-system is optimised for doing a search from the users perspective, finding a file takes log(files) steps and finding related files is trivially cheap. It is likely hard to outdo that with a relational model.

The filesystem is used by different people for different purposes. Any attempt to make sense about “what makes sense” for files is going to be colored very differently depending on what perspective you have.

From an end-user perspective, files are documents that I create and I want to be able to find them in different ways.

From the perspective of a typical app developer, the filesystem is a hierarchical key-value store.

The perspective of a database developer, backup software developer, system administrator, etc. is going to be completely different yet.

It is easy to do that though - set up a database. I've met one, maybe 2, people who don't use file systems the conventional way. They're rare and they generally just want a search index as opposed to relational data.

> It is easy to do that though - set up a database.

I think we have really, fundamentally, failed to communicate here. I’m 100% sure we are talking about different things.

I would see it more as a different and better way to store the same and additional information. In a RDBMS an index is already storing references in a tree format. If you map that to files and folders you now have basically the same thing as a file system.

Doing it more as a DB enables the OS to use the knowledge from RDBMS for efficiency, which I'm sure rivals the best file systems and it's possible to create multiple indexes and views for other use cases.

Our current view on file systems and the knowledge we have is heavily influenced by slow spinning disks, while RDBMS have leveraged RAM a lot more. With todays fast SSDs the file system operates in a reality that is more like RAM than a slow spinning disk.

> In a RDBMS an index is already storing references in a tree format. If you map that to files and folders you now have basically the same thing as a file system.

Yeah, a file system stores general data so it is very easy to map it to an RDBMs that also stores general data.

But what this is identifying that once the relational nature of the data isn't a factor, the best lookup structure is a tree.

You don't need the relational aspect per se, but if you have indexes by filename, filetype, creation date and size (which I do not find unreasonable). It becomes trivially fast to ask the question:

> Give me all image files > 100 KiB from 31. December 2021 to 1. January 2022

While in current file systems you need to scan the entire content of file metadata to get that information. It will take a long time, especially if you have a lot of small files (think Windows C: drive).

That might not be something the average user would do by themselves, but developers of, say, image processing apps certainly would.

Might be the next big thing. But if that is a valuable feature it is _really_ easy to implement. App, sqlite, scan file system once, build up a database of metadata then put 3 indexes on it. I expect a lot of photo apps can already do that.

But all that still isn't really leveraging the power of the relational model. That is simply indexing files on a lot of different attributes, ie, leveraging an RDBMS implementation detail where they use trees. The point of the relational model is relational algebra (SELECT, JOIN & WHERE in SQL terms). And WHERE isn't the interesting one out of those 3, it is JOIN.

If the use case for a relational filesystem is interesting filters then it sounds a lot like a false start.

Of course it's simple to do for every app. But they have to keep that database updated. Either by running a service and listening to all changes to the file system or by running it from scratch every so often. Neither is a good idea from a performance point of view.

I can see where you're coming from with the false start if looking at it from that isolated point of view. I see it more as the first step to storing data, in general, in an RDBMS and once applications start to utilize that, new use cases will start to emerge. Linux in particular with it's "everything is a file" philosophy seems to suited to use this model. A table for processes, files, network connections, ect.

Go meta, and consider projecting distinct hierarchies on document collections. For example, I may have a hierarchical view of my documents organized by content matter, and another one organized by projects, etc. So there is your JOIN.

The relational model also covers transactions. You could imagine opening multiple files, writing data to them, and rolling back a.k.a. reverting to the old situation if something goes wrong. This would protect against file corruption, and would also be interesting for e.g. installers. E.g. a transaction could cover all file changes, registry updates, etc.

Windows did this for a short while, but I believe it was removed again. I don't know the exact reason.

The relational model isn't logically related to transactions or rollbacks. It is easy enough to have those without the relational model.

The relational model is orthogonal to transactions. Ted Codd literally called out consistency challenges in the paper. Transactions were not invented then.

"Finding files" has never been my problem. Organizing and categorizing them has. And the reason it's been a problem is because my files don't contort into a neat hierarchy.

> I'm not immediately seeing how that would be useful in a file context, where generally you want to look up a specific blob of data (aka a file).

I would really enjoy a filesystem that's a loose tag-based hierarchy rather than a strict single folder tree.

In SQL terms, something like:

   create table files (
       file_id bigint primary key
     , file_name text
   )

   create table tags (
       tag_id bigint primary key
     , tag_name text not null
     , parent_tag bigint null
     )

   create table file_tags (
       file_id bigint
     , tag_id bigint
   )

So tags are organized in hierarchies, but there can be multiple parallel hierarchies, and a file can belong simultaneously to multiple hierarchies. Say one is a flat list of tags by user, another is a flat list of tags by apps, and another represents replication or backup strategies, plus the usual directory organization.

(I'm not sure if a file should be allowed to belong to multiple tags within the same hierarchy; my gut says no.)

Let's say that as a convention we separate the root tag in each hierarchy with a colon, and other tags with a slash. Your file 'cool_code.py' may be found under the tags 'storage:sda', 'users:roenxi', 'apps:pycharm', 'pycharm:projects/cool_app/src/utils', and 'rclone:gdrive/roenxi@gmail.com/202201'.

This looks really cool!!

> (I'm not sure if a file should be allowed to belong to multiple tags within the same hierarchy; my gut says no.)

Gmail is a good example. Has very flexible tagging schema. Every tag exists in a hierarchy. Emails can have multiple tags.

SQL is not good for hierarchies though. You need to use recursive CTEs or denormalize relations and things.

> A tree-based file-system is optimised for doing a search from the users perspective, finding a file takes log(files) steps and finding related files is trivially cheap. It is likely hard to outdo that with a relational model.

A tree structured file system is effectively conflating an index with relations. Another way to look at it is that most FSes are like databases which only allow one index, and expose the index to users.

What is the equivalent of a directory, and of a file? From the relation point of view, what would your tables be?

It doesn't map that well, the idea sounds intriguing, but in practice a filesystem seems like a better model for files.

You would not have a "directory" in the sense you use the word (why would directory be required in a FILE system?)

Instead a heap of files and you search for them using metadata (e.g. each file has an associated key value store that you search on).

Music catalogues is a good example. In the 90s one vould wonder if the music sitting on HD should be organized as "Music/(genre)/(artist)" or "Music/(artist)/(genre)" as directories. Different choices was best for different persons.

Eventually many music players for desktop (e.g. iTunes) just made a different metaphor. And Spotify, Netflix etc do not use a hierarchy but you search for items using metadata.

Another example is e.g.

    /lib/libfoo-1.2.so

Where one could instead have

    libraryname=foo
    scope=system
    version=1.2
    sharedlibrary=true

Or similar

> Music/(artist)/(genre)

Well that's only necessary when your music library contains Zappa.

That's not relational, that's closer to a NoSQL store.

That is orthogonal, NoSQL can be used as a relational DB too. SQL is one way to do a relational DB.

Also what I describe is close to SQL if e.g. you declare a set of properties and property types for each "filetype".

Relational is one aspect of a DB.

Schemaless or not is another aspect.

A "file" is just a key-value entry where the "key" is some label in an arbitrary namespace, and the "value" is a blob of bytes. The "directories" of a relational FS would be dynamically generated: they would arise as query result sets based on user input, as opposed to being fixed and materialized on-disk. This is how the relational FS worked in BeOS, but the exact same "virtual directory" feature was also ported to some versions of Windows.

> Maybe the File System shouldn’t be hierarchical but rather relational as well

How do you know it isn't?

Reply on news.ycombinator.com