Versity open-sources ScoutFS | Hacker News Reader

Versity open-sources ScoutFS

versity.com

18 pointsfh97312 comments

"The super block is stored in two 4KB blocks at a known location at the start of the device."

There should be more than 2 slots for the super block, e.g. at least 64 slots for the super block, to allow for better wear-leveling for SSDs. AFAIK this has been a problem for some filesystems, which wear out the device due to the high frequency of super block updates.

The latest super block should also be replicated 2-4 times on disk to survive corruption. Otherwise, the loss of a mere two contiguous disk sectors could result in loss of the entire disk.

A super block of 4KB also looks like they might be relying too much on the device to guarantee atomic sector writes (this is almost always the case, but not necessarily), instead of using a cryptographic signature to verify that the super block was written atomically. Also, 4KB seems small these days for a super block, especially if the super block needs to include replicated information relating to RAID sibling devices etc.

If anyone from Versity is reading this, there is a typo in the release:

“We are releaing the technology under the GPLv2 open source license.”

Doh! Thanks for letting us know, will get that fixed. Of course, patches/contributions are always welcome.... :)

What's the difference of archival filesystem vs. distributed filesystem? I assume other distributed filesystems could be used to archive files as well, namely https://github.com/moosefs/moosefs

The focus here seems to be on very high scale, not just capacity but also number of files and scan performance which would be important for an archival store and which are often pain points for other distributed filesystems. Similarly, ensuring data integrity is a high priority for such a system. I certainly wouldn't compare it to a single-metadata-server pseudo-distributed filesystem that doesn't even honor O_SYNC writes.

Couldn't have said it better myself! For more details on requirements as we designed the filesystem and why current filesystems didn't meet our needs, see this blog post: http://www.versity.com/blog/versity-open-sources-scoutfs

What I didn't see in the white paper, is whether the contents of each file are checksummed?

It seems only the metadata is checksummed.

I have used ZFS and the ability to run a check on the entire set of metadata and actual files is very reassuring (though it is definitely time consuming).

Both metadata and data are checksummed. A unique feature of the filesystem is that it doesn't trust what the storage tells it, it actually verifies it. Thanks for pointing this out - we'll get these details in the next version of the paper.

That is great to hear and I look forward to reading more about it, thanks.

Looks interesting, and apparently it's really a filesystem unlike so many thing that just slap "FS" on the name and call it a day. tears of joy White paper etc. is a bit low on details, though.

Linux mainline soon?

Definitely.

Reply on news.ycombinator.com