Hacker News Reader

Comment by londons_explore | Hacker News Reader

Is a 64MB object size really typical of the usecase most people will be using this for?

I would imagine 1kb objects to be far more common...

At 1kb or less I would likely store as blob in traditional RDMS or noSQL store. It is not worth the extra network call at 1 KB size to use object storage.

I suppose it depends on the type of data? If those are e.g. video files, I don't suppose there are many 1kB files...

Aha, I replied above and you are saying the same thing!

Small objects, 4K ~ 4MB, are very common in machine learning area(audio, video, text), or surveillance, game assets, etc. SeaweedFS saw a lot of usage in these areas.

No, object store works better with a small number of large object than with a large number of small object.

1kb object would be terribly inefficient if you have too many of them. It's slow even with regular file system, copy 1 million 1kb files would take forever. For small objects, it's best to pack them into larger block before storage, like how a database does with its data.

Actually SeaweedFS does just that, append small objects into larger blocks during write or update, and run garbage collection during compaction.

SeaweedFS sounds very interesting indeed, though I'm a bit concern about the complexity. For example, there are so many Filer metadata backend options that it's difficult for me to pick one.

I think the document should have a clearer guideline about best practices when deploying a SeaweedFS cluster for different situations: small cluster, large cluster, H/A cluster etc..

This is a fair complaint. :)

For filer metadata, you should just pick the one you are most familiar with.

There is a wiki page for production setup. https://github.com/chrislusf/seaweedfs/wiki/Production-Setup

That's common for partioned data. If you have a couple hundred GB table or log you can split it into a bunch of smaller blocks for parallel processing. I think 64MB tends to be a good medium between overhead of tons of tiny files and inability to parallelize/shard with fewer large ones

Possibly a difference in terminology. Minio is in the same space as S3, Ceph, and HDFS - sometimes called blob store or block store. Each object might be part of a log, video or other large file/stream.

I would call an object store in which the objects are mostly < 1 kiB a key value store.

What are you storing in an object store that fits into 1kb?

64MB is far, far closer to our usecase than 1kb. I don’t even think our schema files/metadata which are arguably the smallest part fit into 1kb.

Reply on news.ycombinator.com