Practical File System Design: The Be File System (1999) [pdf] | Hacker News Reader

Practical File System Design: The Be File System (1999) [pdf]

nobius.org

101 pointse-sushi23 comments

I remember reading about the BeFS, and how advanced it was for its time. Can anyone who does filesystems help me remember why it was so powerful?

Is anything out there now comparable? Have we surpassed BeFS? Any residual functionality that contemporary filesystems lack?

It had a lot of features for the time (64-bit, journaling, etc.), but one of the most used features in BeOS was that arbitrary metadata could be attached to files, exposed in Tracker or other applications, and could be used to search for files. One of the examples of the time were scripts to take MP3 ID3 tags and store them as metadata. Email was stored as individual files on the file system, and the email client was just a Tracker window with columns displaying email relevant metadata.

Spotlight on OS X supports similar metadata storage and querying, but it's built on top of HFS+, not directly into the file system (and surprise, was built by Dominic Giampaolo, who worked on BeFS, author of the linked book).

WinFS claimed a lot of similar features shortly after Be was acquired, but to my knowledge, no file system has provided similar features _and_ had spextensive application support to leverage them.

I think the clever bit is when a user sees their email in the file browser[1] they can mentally connect with that and start using their file system as the database that it is, building their own dynamic todo list[2] that isn't managed by any application!

Almost every filesystem "supports metadata", and while I agree that application support could be better, I think the real key is getting users to use that space themselves.

[1]: https://birdhouse.org/beos/refugee/bemail.jpg

[2]: https://birdhouse.org/beos/refugee/trackerbase.gif

BeFS's metadata support was much different from usual—it was a key/value map attached to each file. NTFS supports extended attributes and alternate data streams, which you could use to do the same thing, but none of the *nix filesystems have anything like that. What sort of metadata do you mean with “almost every filesystem ‘supports metadata’”?

JFS, XFS, ReiserFS, ext3/4, and btrfs all support extended attributes – that's become a bigger deal for things like ACLs – but along with varying size limits the share the same limitations as NTFS, HFS+, etc. of not indexing those values.

That means there's no way to do a query like "file type = audio/mp3" without either walking the filesystem or querying a separate index, which removes a lot of the benefits.

I think NTFS can index alternate data streams¹, but yeah, I agree no filesystems are on the level that BeFS was of easy querying and versatility.

¹ I am not sure about this, but https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... seems to suggest it

I'm not sure either – it definitely did not in the past but I haven't needed to look into it for years. Things like WinFS were going to solve that the right way but since that never shipped I assume they were still relying on a bolt-on indexing service.

I loved BeOS. I bought and ran versions 4.5 and 5.0 Pro. I also owned a copy of Scot Hacker's BeOS Bible; which I read front to end. I visited the BeNews and BeOS Tips websites almost everyday.

As for BeFS itself, it was a pretty awesome FS. I remember I used to use a utility called MP3 Army Knife, which could copy MP3 metadata to BeFS attributes. I then used Marco Nelissen's SoundPlay (shareware which I bought) to play them; basically using Tracker or BeFS queries to generate playlists.

I miss that OS.

You've likely heard of this before but Haiku[0] is a modern open source continuation of BeOS, which is almost usable.

[0]: haiku-os.org

And the metadata support was instantaneous, without noticeable overhead or delays. I mean, your query windows would live update as you would expect them to. Despite faster machines, this doesn't feel as instant on Linux, OS X or Windows 7 (haven't used 8 or 10).

> And the metadata support was instantaneous, without noticeable overhead or delays

Seconding things: BeFS was staggeringly fast – doing a simple query in something like Outlook or Mail.app is still nowhere near the same class as either the standard BeOS mail client or PMMail, even with generations of hardware improvements. On an idle system with an SSD, a Spotlight or Windows Search / Cortana query isn't as responsive as a BeFS query was on my old Pentium 90 with 5200 RPM drives.

At least some part of that is due to the fact that BeOS had a very good scheduler – I remember being impressed one time when I was transferring a lot of video off of a camera[1] building Mozilla from source, and realized that it didn't make either web browsing or email less responsive.

1. this was DV over Firewire so it wasn't a hard-realtime task but the tape made it soft-ish, and the sound change was quite noticeable when the tape drive had to spin down to wait for the host PC, which happened under Windows but never BeOS.

Looks like this has been posted a few times over the years. I wonder how much is new.

Previous discussions: https://news.ycombinator.com/from?site=nobius.org

I read the book 10 years ago, so not much. But in any case it's a great book.

As I seem to recall the author now works for Apple and worked on their new file system.

Yep. According to Adam Leventhal, he was the lead engineer of APFS at Apple [0].

[0] http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/

It's funny to see a book on "practical" filesystem design make absolutely no mention of FAT, even in its chapter on other filesystems, despite it being probably the most widely supported read-write filesystem --- or family of filesystems --- in existence.

Because FAT is a very impractical file system

(if you read the section on "Other file systems" it's clear they're picked for being interesting case studies - I imagine FAT is simply not interesting to discuss)

Because FAT is a very impractical file system

That apparently doesn't seem to matter as it is supported almost everywhere... they've picked some of the most featureful ones, but I think there is much to be said for simplicity and ubiquity since that is what really enables data interchange.

Then again, the fact that it calls BeFS "practical" reminds me of that old joke about maths textbook titles.

> > Because FAT is a very impractical file system

> That apparently doesn't seem to matter as it is supported almost everywhere... they've picked some of the most featureful ones, but I think there is much to be said for simplicity and ubiquity since that is what really enables data interchange.

That's true for filesystem choice, but the ubiquity of FAT has nothing to do with "Practical [or impractical] Filesystem Design".

You can't sit down and design a filesystem with "installed everywhere" as a feature; things which are everywhere have already been designed, new designs aren't installed anywhere to begin with.

FAT is not very interesting from a filesystem design perspective (ask someone who doesn't know much about filesystems to try to design one; chances are it will look like FAT¹), so that hardly surprises me. Anyone who buys a book about filesystem design and implementation will already by quite familiar with FAT.

¹ Which is why FAT is so ubiquitous—it's dead simple.

This is a fantastic book, but do we have the rights to it? What is this URL?

It's the author's website, I assume yes.

Yes, the author published openly a few years ago, when it got out of print. Actually, the link is to the authors site: http://www.nobius.org/~dbg/index.html

Reply on news.ycombinator.com