SeaweedFS – A simple and highly scalable distributed file system

The README mentions that Seaweed is much easier to set up than Ceph. The more apt comparison would be the RADOS component of Ceph--both Seaweed and RADOS are key/value oriented. I've now set up both, and for sure Seaweed is easier to get going.

I have a fair bit of experience with RADOS (all quite positive) but I really do appreciate the ease of use Seaweed presents. I've only been using it for an hour or so, just doing benchmarks and walking through fault scenarios, and so far Seaweed looks quite good!

FWIW, I don't think this should be called a filesystem at all. Seaweed isn't a filesystem which forsakes some aspects of POSIX, it's nothing like a filesystem. The more proper term would be "distributed object store."

My original plan is to break normal file system into composable pieces, and make the file system more scalable.

The first piece is the key~blob storage, which consists of master and volume servers. The second piece is the filer server.

The first piece should be self contained and agnostic of the layer above, yet still easy-to-use in normal cases where only key~blob service is needed.

Hell, even network filesystems (non distributed) aren't where they should be.

To illustrate: there is no fundamental reason why a distributed filesystem shouldn't, in practice, perform more or less equivalently to a network, non distributed filesystem. That's not the case today; existing distributed filesystems have slowpaths - generally, things involving metadata and anything that requires coordinating servers - where they're drastically slower than say existing NFS or 9p implementations, and these aren't obscure corner cases, these are things real applications hit.

Reasoning from first principles, we know that: - It's not going to be possible to create a distributed filesystem that doesn't have these slowpaths, it's probably always going to be possible to create an evil/pathalogical workload that requires lots of cross server synchronization, but: - Most real world applications don't behave so badly. Imagine you just want a single filesystem that has a crapton of home directories, and you want to throw it all on a distributed filesystem instead of giant Netapp filers or whatever. We should be able to do that, and have it perform just as well (to with epsilon) as using NFS and explicit sharding. We can't really do that today.

In my opinion, we don't really have an architecture implemented today that's capable of solving these problems - that said, I'm not familiar with Ceph's architecture so I don't know how it tackles sharding. I have my own ideas about how it could be done (for awhile, I was working on something on top of the bcache/bcachefs codebase; I actually implemented working, fast distributed block storage, but not filesystem, and it didn't have nearly everything implemented that it would have needed to scale out horizontally very far). But I never got far enough into the filesystem aspect to really prove anything.

For network (non distributed) filesystems - again, those aren't as good as they ought to be either. But the problem there is much simpler - nobody's built anything to my knowledge that would let you really do cache coherency correctly: you can't do it by just using an existing local filesystem implementation and serving it over a new protocol, because you can't do cache invalidation correctly without storing fine grained versioning information that local filesystems don't need, and nobody's wanted to try grafting that onto existing local filesystems. A lot of the story of NFS is hacks to get around that problem.

https://github.com/chrislusf/seaweedfs#rack-aware-and-data-c...

(Not an author or anything, just found this project today, and it's interesting enough for me to test it out.)

Edit to add: I've done some testing, SeaweedFS definitely replicates. When the client wants to read from a volume, it should do a lookup and Seaweed will return the URLs of all volume servers which are up at the moment. So far I've just been testing 2-way replication with one server down at a time; it works fine.

> Also - what's the protection mechanism for files? If there is none, I literally have no idea what the purpose of this FS is. Under what circumstances would I need a giant slow data storage tier that has no redundancy to speak of?

Caches, that's what makes me interested in this (having to cache lots of images). We're using mogilefs for this currently.

TLDR, Haystack implements a sort of append-only journal design where smaller files are concatenated into a single large "journal" file. After adding, you get a "filename" back that is based on the information needed to locate the data you are interested in. volumename_seekposition.jpg, for example.

Essentially, you eliminate the need for FS metadata this way, which leads to less disk hits for read operations. Open volume, seek to position, read contents (including metadata) until end of file, you're done.

There is then a single-master database that controls the location of the volumes (with a redundant failover), which is possible because managing volumes is less difficult than managing the position of every single file in all the volumes in the system. I think they just use MySQL, amazingly.

Pretty neat idea. It's designed very specifically for photo storage at Facebook.

The main drawback with this design is that it doesn't support binding filesystem paths well - to do that you need a K/V database to match the name you want to the filename needed for Haystack/SeaweedFS. SeaweedFS can do this with a separate Redis and Cassandra "filer" database, but the Redis and Cassandra backends are currently implemented as "flat namespace" stores, so filers using them may not perform directory listings at this time. Huge drawback if you need to 'ls' subdirectories of files, for example.

That was a dealbreaker for me for neocities.org implementation, also maintenance was still an open question with SeaweedFS (when things fail, what do I do to fix them?) so I ended up going with CephFS.

    "Instead of managing chunks, SeaweedFS manages data
    volumes in the master server. Each data volume is size
    32GB, and can hold a lot of files. And each storage node
    can have many data volumes. So the master node only needs
    to store the metadata about the volumes, which is fairly
    small amount of data and is generally stable."

If the client crashes, the writes would fail.