DBOS: A DBMS-oriented Operating System [pdf] | Hacker News Reader

DBOS: A DBMS-oriented Operating System [pdf]

vldb.org

80 pointsmatt_d21 comments

There was a small thread recently (we invited matt_d to repost):

DBOS: A DBMS-oriented Operating System [pdf] - https://news.ycombinator.com/item?id=29803799 - Jan 2022 (6 comments)

This idea has been around for a while... BeOS allowed for indexing and metadata which allowed you to use the file system a lot like a relational database. It was a really cool feature and made it easy to write software that stored tabular data. MS tried back in the Windows Longhorn era... and gave up on it. I think there's more than one 70s and 80s era big iron OS that did this, too.

The file system has been really hard to get rid of - it hides a surprising lot of details on how bits get read and written from hardware, and that hardware has changed a lot in the past decades. SQLite is a great codebase to look at to see what really goes on there.

Filesystem is small aspect here, the big deal here is putting stuff like task scheduling and IPC on top of DBMS. And even for files, the point here is more on having distributed filesystem as first-class citizen and less on being able to do sql-like queries.

Although not the same as what the paper talks about, an approximation would be Oracle running with raw volumes managed by the DB, on a cluster, using Web based UIs done on Apex.

I'm toying with the same idea but also with the idea of treating files and folders as the same thing: nodes in the filesystem.

It's an interesting idea but coding it is somewhat hard as I also want decentralized synchronization.

DBOS was presented at the CIDR conference a couple weeks ago: https://youtu.be/L4LsYda9hKU

I am sure there are a lot of details that I missed, but this sounds a lot like an IBM i series. If this is free software I can see a market for it. If this is commercial software then . . . well I don't know good luck.

briefly brought back memories of the PICK OS https://en.wikipedia.org/wiki/Pick_operating_system which ran our council tax system back in the 90s when I was an IT Manager <shiver>

I remember back in the day, overly Unix people had a mantra, a filesystem is not a database which is kind of a non-sentence like a Macintosh is not a typewriter, also common at the time.

The dominant storage technology then, disks, had structure and performance was determined largely by that structure. Ignoring that structure did not solve the performance problem. I thought that maybe a filesystem actually needed a schema and wrote a semester paper for that for CS262.

Well that went over like a lead balloon. It's less necessary now with SSDs, but I often think that Unix/Linux oversimplifies and then pays technical debt for decades for having adopted a simpler approach than VMS/...

And the success of VMS in the modern era speaks to that.

Some of the structures of Files-11 lives on in NTFS in part. Some of the structure of VMS lives on in NT.

It is just another example of all the things UNIX people decided to ignore, yet thanks to Worse is Better, now it is everywhere with plenty of technical debt to fix.

Whenever I read a HN comment and recoil because it's something that I could well have written myself, inevitably, it was @pjmlp. :-D

Seems obvious to me that a file system is a type of hierarchal k/v database.

Maybe one day we will finally get orthogonally persistent os that is not niche: http://tunes.org/wiki/orthogonal_20persistence.html

I used to be a fan of the idea, but over time I formed the opinion that "ordinary" software is just too error-prone to be worthy of full orthogonal persistence and life-long up-times. Tunes was supposed to be based on a high-level type-safe, memory-safe language runtime — but that would just be the least requirement. IMHO, formal methods and proof-carrying code (or some other novel idea altogether) would have to be applied throughout to avoid errors that would persist.

Orthogonal persistence has got an up-swing in academia in recent years after the introduction of new non-volatile memory tech, such as 3D CrossPoint. But I think that some of these researchers have applied it a bit naïvely. The best NVRAM is still much less reliable than DRAM, so it still needs e.g. wear-levelling and error-correction (and even write-throttling) — like what modern file-system stacks do.

Feels like someone took hadoop and osquery and smashed them together. And like hadoop it feels like overkill if you don't have a large cluster of computers to distribute work on.

reminds me a lot of IBM OS400.

Maybe Google can add a NFS frontend to Spanner, I wonder what that would look like.

Probably would be better to try this as a layer on top of FoundationDB.

It was kind of designed to be a distributed systems building block that you can develop interfaces on top of without worrying much about implementing the durability, consistency, or distribution of data.

These kinds of ideas always remember me of WinFS.

Reply on news.ycombinator.com