Hacker News Reader

Comment by vog | Hacker News Reader

How does this compare with the already popular distributed file systems IPFS?

IPFS is a protocol and implementation of the distributed web. SeaweedFS is an open source version of Facebook's Haystack, a centralized filesystem (with redundancy across multiple servers). See https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...

TLDR, Haystack implements a sort of append-only journal design where smaller files are concatenated into a single large "journal" file. After adding, you get a "filename" back that is based on the information needed to locate the data you are interested in. volumename_seekposition.jpg, for example.

Essentially, you eliminate the need for FS metadata this way, which leads to less disk hits for read operations. Open volume, seek to position, read contents (including metadata) until end of file, you're done.

There is then a single-master database that controls the location of the volumes (with a redundant failover), which is possible because managing volumes is less difficult than managing the position of every single file in all the volumes in the system. I think they just use MySQL, amazingly.

Pretty neat idea. It's designed very specifically for photo storage at Facebook.

The main drawback with this design is that it doesn't support binding filesystem paths well - to do that you need a K/V database to match the name you want to the filename needed for Haystack/SeaweedFS. SeaweedFS can do this with a separate Redis and Cassandra "filer" database, but the Redis and Cassandra backends are currently implemented as "flat namespace" stores, so filers using them may not perform directory listings at this time. Huge drawback if you need to 'ls' subdirectories of files, for example.

That was a dealbreaker for me for neocities.org implementation, also maintenance was still an open question with SeaweedFS (when things fail, what do I do to fix them?) so I ended up going with CephFS.

Thanks for the write up! You are correct on most of the points.

There are postgres backed filer added already, which can list files.

That's great news! I'm going to take a look at it tonight.

Postgres of course does needs some special work to make it failover in production gracefully to a standby server. It would be nice to see something that's designed for it like redis work, but of course, that's a limitation of how redis is designed.

Regardless, I don't think I mentioned this properly, but SeaweedFS is amazing and it's very impressive, and using it at some point is not yet off the table for us. :)

Actually there is a pull request to add redis support to list files. https://github.com/chrislusf/seaweedfs/pull/254

I just haven't have time to test it and make it work yet. If you want to, you can help to verify this.

If that worked, it would be quite the game changer.

I could play with it a bit, but a full code analysis would be over my head.

Can I incentivize you in any way to review it? :)

This isn't truly distributed in that sense:

    "Instead of managing chunks, SeaweedFS manages data
    volumes in the master server. Each data volume is size
    32GB, and can hold a lot of files. And each storage node
    can have many data volumes. So the master node only needs
    to store the metadata about the volumes, which is fairly
    small amount of data and is generally stable."

So there's no replication?

There is. I am not following your reasoning.

i think he's asking if there's replication for the data stored on the master in case the master goes down, i.e. can there be multiple masters?

You can (should) have multiple masters. But each volume has a primary master. I did see other masters loose the volume state information after the primary master has gone down and been taken out of the cluster.

Good observation. I will make the recovery much faster with gRPC calls, which can detect broken network much faster than http heartbeats.

The master nodes only have soft states. All actual states are stored on volume servers, together with the actual data.

If one master node goes down, the other master nodes will elect a new leader and control the system.

I am not sure that SeaweedFS and IPFS fit the same use case, IPFS seems like it wants to a bedrock layer of a newer style of internet. SeaweedFS is more of a Blob store. So comparing them might be possible but also might not be the right idea.

For one, SeaweedFS states that it is not POSIX-compliant.

It's not a filesystem either...I don't get why they keep using that terminology. It doesn't have directories, permissions, seek(), copy without the client pulling all the data and pushing it back, etc.

It's a KV data store. Which is fine...nothing wrong with that. It just isn't a filesystem in any sense I'm familiar with.

Edit: Ahh, I see. There's an optional piece that does provide at least directories. Still short of being a "filesystem" IMO, though: https://github.com/chrislusf/seaweedfs/wiki/Filer

There are many ways to build a filer layer. Current implementation is just one way to do it.

Adding POSIX or not is more of a design choice. The blob storage layer can support both ways.

Right. The word filesystem has a defacto meaning for many, though. Not POSIX either...FAT, NTFS, etc are filesystems. There's implied functionality that goes with the word.

This isn't a filesystem anymore than etcd would be.

Reply on news.ycombinator.com