Hacker News Reader

Comment by weinzierl | Hacker News Reader

We've been there. Before we had filesystems as we know them today, there were many different ways of persistent data storage. Roughly these could be grouped into two camps: The files camp and the records camp.

The record based approach had many properties we know from modern databases. It was a first class citizen on the mainframe and IBM was its champion.

In my opinion hierarchical filesystems won as everyday data storage because of their simplicity and not despite it. I think the idea of a file being just a series of bytes and leaving the interpretation to the application is ingenious. That doesn't mean there is no room for standardized OS-level database-like storage. In fact I'd love to see that.

View on HN

I've always been annoyed how searching a few hundred thousand NTFS records for a filename containing some arbitrary text takes a relatively long time - even using specialized tools like FileLocator Pro which I believe directly scan low-level structures like the MFT - while I can do an equivalent search in a SQL database in milliseconds. I wish filesystems like that one had vastly more performant indexing structures for the metadata (without relying on add-on layers that defer indexing and - at least in my experience - tend to break down or be out of date or obfuscate files they think you don't care about - I'm looking at you, Windows Search).

I've been using a free tool by voidtools tool called "Everything" for years, it provides almost-instant search on windows NTFS volumes: https://www.voidtools.com/support/everything/

Thanks. I used it years ago but it didn't suit my needs. I haven't found an indexing tool that does.

It's the difference between synchronous indexing that's baked into the system (as in file system metadata structures and database indexes, which update at the same time your data is changed) vs. fragile add-ons that index asynchronously (which in general I find tend to be too slow to update, missing results, and prone to breaking).

well, there is a good reason they cant do that. they dont know what the files are.

perhaps filesystems should be extensible in a way that supports indexing intelligently.

(i know you mentioned an aversion to addons)

The lack of structure is ultimately why writes to filesystems are comparatively fast and reliable, though.

Modern filesystems in a way combine both approaches - they store the data unstructured but give the ability to also store metadata (attributes) in a structured way.

That functionality is mostly an afterthought in ext and ntfs, though. More of a big deal for Apple with hfs but still not something you'd build a database on.

> I think the idea of a file being just a series of bytes and leaving the interpretation to the application is ingenious.

the file as an opaque box for applications to store a real data structure is poisonously anti-file. it's totally what files are, what we think of them, but imo, systems like 9p, or linux's procfs or sysfs are The True Way for files: small discrete pieces of data which are part of a system of directories tlthat express a larger compilated hierarchical system of data.

Files won, but only the stupidest wrongest version. Easy to copy and manage but utterly useless on their own, unscriptable, pointless eithout their complex applications there to use them.

I dont think db's/records are that interesting either. i think we just need to really try files. Fine grained files. As opposed to these big ole blobs the OS cant really interact with.

Don't you need mandatory locking of files and directories, or rather powerful transactional semantics for the filesystem then?

A lot of filesystems have snapshots. NTFS & others have transactional capabilities. I don't regard locking as necessary or helpful when the OS can provide these capabilities.

There aren't really any hurdles to implementing an SQL database on top of a plain block device, are there? So I wonder why no-one has gone there. This would allow the database server to do caching in a way that makes sense for the database and not have to hope that the filesystem cache does the right thing.

Commercial databases routinely do exactly that. O_DIRECT basically exists because Oracle needed it

Reply on news.ycombinator.com