Hacker News Reader

Comment by roenxi | Hacker News Reader

The only thing the relational models really gives you is the ability to join two relations, which lets data decompose for storage but re-compose for many different uses. I'm not immediately seeing how that would be useful in a file context, where generally you want to look up a specific blob of data (aka a file).

A tree-based file-system is optimised for doing a search from the users perspective, finding a file takes log(files) steps and finding related files is trivially cheap. It is likely hard to outdo that with a relational model.

View on HN

"Finding files" has never been my problem. Organizing and categorizing them has. And the reason it's been a problem is because my files don't contort into a neat hierarchy.

> I'm not immediately seeing how that would be useful in a file context, where generally you want to look up a specific blob of data (aka a file).

I would really enjoy a filesystem that's a loose tag-based hierarchy rather than a strict single folder tree.

In SQL terms, something like:

   create table files (
       file_id bigint primary key
     , file_name text
   )

   create table tags (
       tag_id bigint primary key
     , tag_name text not null
     , parent_tag bigint null
     )

   create table file_tags (
       file_id bigint
     , tag_id bigint
   )

So tags are organized in hierarchies, but there can be multiple parallel hierarchies, and a file can belong simultaneously to multiple hierarchies. Say one is a flat list of tags by user, another is a flat list of tags by apps, and another represents replication or backup strategies, plus the usual directory organization.

(I'm not sure if a file should be allowed to belong to multiple tags within the same hierarchy; my gut says no.)

Let's say that as a convention we separate the root tag in each hierarchy with a colon, and other tags with a slash. Your file 'cool_code.py' may be found under the tags 'storage:sda', 'users:roenxi', 'apps:pycharm', 'pycharm:projects/cool_app/src/utils', and 'rclone:gdrive/roenxi@gmail.com/202201'.

This looks really cool!!

> (I'm not sure if a file should be allowed to belong to multiple tags within the same hierarchy; my gut says no.)

Gmail is a good example. Has very flexible tagging schema. Every tag exists in a hierarchy. Emails can have multiple tags.

SQL is not good for hierarchies though. You need to use recursive CTEs or denormalize relations and things.

I would see it more as a different and better way to store the same and additional information. In a RDBMS an index is already storing references in a tree format. If you map that to files and folders you now have basically the same thing as a file system.

Doing it more as a DB enables the OS to use the knowledge from RDBMS for efficiency, which I'm sure rivals the best file systems and it's possible to create multiple indexes and views for other use cases.

Our current view on file systems and the knowledge we have is heavily influenced by slow spinning disks, while RDBMS have leveraged RAM a lot more. With todays fast SSDs the file system operates in a reality that is more like RAM than a slow spinning disk.

> In a RDBMS an index is already storing references in a tree format. If you map that to files and folders you now have basically the same thing as a file system.

Yeah, a file system stores general data so it is very easy to map it to an RDBMs that also stores general data.

But what this is identifying that once the relational nature of the data isn't a factor, the best lookup structure is a tree.

You don't need the relational aspect per se, but if you have indexes by filename, filetype, creation date and size (which I do not find unreasonable). It becomes trivially fast to ask the question:

> Give me all image files > 100 KiB from 31. December 2021 to 1. January 2022

While in current file systems you need to scan the entire content of file metadata to get that information. It will take a long time, especially if you have a lot of small files (think Windows C: drive).

That might not be something the average user would do by themselves, but developers of, say, image processing apps certainly would.

Might be the next big thing. But if that is a valuable feature it is _really_ easy to implement. App, sqlite, scan file system once, build up a database of metadata then put 3 indexes on it. I expect a lot of photo apps can already do that.

But all that still isn't really leveraging the power of the relational model. That is simply indexing files on a lot of different attributes, ie, leveraging an RDBMS implementation detail where they use trees. The point of the relational model is relational algebra (SELECT, JOIN & WHERE in SQL terms). And WHERE isn't the interesting one out of those 3, it is JOIN.

If the use case for a relational filesystem is interesting filters then it sounds a lot like a false start.

Of course it's simple to do for every app. But they have to keep that database updated. Either by running a service and listening to all changes to the file system or by running it from scratch every so often. Neither is a good idea from a performance point of view.

I can see where you're coming from with the false start if looking at it from that isolated point of view. I see it more as the first step to storing data, in general, in an RDBMS and once applications start to utilize that, new use cases will start to emerge. Linux in particular with it's "everything is a file" philosophy seems to suited to use this model. A table for processes, files, network connections, ect.

Go meta, and consider projecting distinct hierarchies on document collections. For example, I may have a hierarchical view of my documents organized by content matter, and another one organized by projects, etc. So there is your JOIN.

> A tree-based file-system is optimised for doing a search from the users perspective, finding a file takes log(files) steps and finding related files is trivially cheap. It is likely hard to outdo that with a relational model.

A tree structured file system is effectively conflating an index with relations. Another way to look at it is that most FSes are like databases which only allow one index, and expose the index to users.

The filesystem is used by different people for different purposes. Any attempt to make sense about “what makes sense” for files is going to be colored very differently depending on what perspective you have.

From an end-user perspective, files are documents that I create and I want to be able to find them in different ways.

From the perspective of a typical app developer, the filesystem is a hierarchical key-value store.

The perspective of a database developer, backup software developer, system administrator, etc. is going to be completely different yet.

It is easy to do that though - set up a database. I've met one, maybe 2, people who don't use file systems the conventional way. They're rare and they generally just want a search index as opposed to relational data.

> It is easy to do that though - set up a database.

I think we have really, fundamentally, failed to communicate here. I’m 100% sure we are talking about different things.

The relational model also covers transactions. You could imagine opening multiple files, writing data to them, and rolling back a.k.a. reverting to the old situation if something goes wrong. This would protect against file corruption, and would also be interesting for e.g. installers. E.g. a transaction could cover all file changes, registry updates, etc.

Windows did this for a short while, but I believe it was removed again. I don't know the exact reason.

The relational model isn't logically related to transactions or rollbacks. It is easy enough to have those without the relational model.

The relational model is orthogonal to transactions. Ted Codd literally called out consistency challenges in the paper. Transactions were not invented then.

Reply on news.ycombinator.com