Hacker News Reader

Comment by andix | Hacker News Reader

Wouldn’t it be better to keep such files inside a database?

https://www.sqlite.org/fasterthanfs.html They're selling it quite well here, though integration into other products might be the hard thing.

Add https://github.com/guardianproject/libsqlfs

I literally discovered this just the other day, the whole concept is extremely intriguing but it hasn't been updated in some time (I'm guessing some forks have been though).

Relatedly, it would be pretty cool to be able to build your own filesystem features as layers on each other (such as block-level compression or encryption, block-level forward-error correction, cache promote, striping or other redundancy etc.). A modular FS with pluggable (but also upgradable) components and some sort of structured feature-flag schema metadata, and a way to migrate between variations (such as switching to a different compression or encryption algorithm).

Some of what you're describing is in ZFS. E.g., it has multiple compression algorithms; you can create many datasets -- each of these is either a ZFS filesystem or a volume which can have another FS deployed to it, but with compression and encryption handled by the "host" ZFS.

You may also be interested in FreeBSD's GEOM subsystem for storage.

In my experience, integration with products is not really that hard. I've used sqlite3 bindings in half a dozen languages over the years, and they are generally fairly robust.

A filesystem is a database.

So what people think of as a database is a database inside a database?

Yeah. It's like that old Xzibit meme.

File systems aren't relational databases, but more like NoSQL half a lifetime before it was cool. They're typically backed by the same data structures and solve the same problems.

Indeed, some database servers (Oracle IIRC) don't use a filesystem, but write to a filesystem partition directly.

But a special kind of database, that is usually bad with a lot of small entries, especially if they don't have a hierarchy (many files in one folder). Most SQL or NoSQL databases are much better at organizing small "files".

The difficult use case isn't many files in a folder, it's just many files period.

Reply on news.ycombinator.com