Hacker News Reader

Comment by hcta | Hacker News Reader

Can you clarify what point you're making? If you're trying to argue that adding an extra layer can only reduce performance, any cache is an obvious exception to that. Are you saying it's extraneous to use a database as a storage abstraction because they have to sit on top on filesystems, and filesystems already exist?

View on HN

The point I'm making (and I'm not certain that it is true) is that ultimately if you want to store raw data, a filesystem seems more likely to be what you want to use. Put differently, BLOBs in the DB end up (necessarily) as blobs on the disk, and managing blobs on a disk is precisely what filesystems are intended for.

But yes, on top of that, there's the question that in the end even the DB will need something very, very much like a filesystem between them and the storage hardware ... which opens up the question whether this should remain hidden to every other application, or whether it makes sense that for certain kinds of applications, they too would use it (i.e. just like today)

> managing blobs on a disk is precisely what filesystems are intended for.

A filesystem is doing much more, e.g. providing naming and management (directories, symlinks, access control, extended attributes, cache management, …) for files for manipulation by humans and applications, whereas RDBMs only need fixed-sized blocks of storage.

Some databases actually support using raw disks without a normal filesystem, which can have advantages by removing the extra layer of abstraction, e.g.:

https://dev.mysql.com/doc/refman/8.0/en/innodb-system-tables...

https://docs.oracle.com/en/database/oracle/oracle-database/2...

https://www.ibm.com/docs/de/db2/9.7?topic=creation-attaching...

I created a kind of object store (https://www.Didgets.com) that originally was designed to replace file systems. It manages the data streams (i.e. blobs) for each object very much like a file system does for each file. Although I have a few algorithms that make allocation and management of all the blocks very efficient, my testing shows almost equivalent I/O speed for reading/writing the data.

It is in the metadata management where my system excels. The table of file records for volume with over 200M files only needs 13GB read from disk and that much RAM to cache it all. Contextual metadata tags can be attached to each object and lightning fast queries executed that use them. The objects (Didgets) can be arranged in a hierarchical folder tree just like file systems use, but they don't need to be.

> But yes, on top of that, there's the question that in the end even the DB will need something very, very much like a filesystem between them and the storage hardware

So the answer to this question is no. The “filesystem” that a relational database uses - ie how it organizes and allocates on the block layer is so different from the DOS/POSIX semantics that you wouldn’t recognize it as a filesystem - so to say it is very, very much like a filesystem is dubious.

Reply on news.ycombinator.com