chrislusf

Should not be a problem.

One similar use case used Cassandra as SeaweedFS filer store, and created thousands of files per second in a temp folder, and moved the files to a final folder. It caused a lot of tombstones for the updates in Cassandra.

Later, they changed to use Redis for the temp folder, and keep Cassandra for other folders. Everything has been very smooth since then.

Thanks for sharing! I work on SeaweedFS.

SeaweedFS is built on top of a blob storage based on Facebook's Haystack paper. The features are not fully developed yet, but what makes it different is a new way of programming for the cloud era.

When needing some storage, just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block.

There will be more features built on top of it. File system and Object store are just a couple of them. Need more help on this.

I work on SeaweedFS. Ping me if you need some help to understand why SeaweedFS is much faster.

It is still growing. This is not the case since many months ago.

Also, welcome to make a PR.

Another research suggests rubbing eyes also increase risk for Alzheimers.

The virus getting into the eyes will slowly eat the brain.

Can someone please critique on the difference between SeaweedFS and Colossus? https://github.com/seaweedfs/seaweedfs

Need some advice on how to make SeaweedFS more scalable.

Thanks for the detailed clarification! I am too deep into the SeaweedFS low level details and am all ears on how to make it simpler to use. SeaweedFS has weekly releases and is constantly evolving.

Depending on your case, you may need to add more filers. UCSD has a setup that uses about 10 filers to achieve 1.5 billion iops. https://twitter.com/SeaweedFS/status/1549890262633107456 There are many AI/ML users switching from MinIO or CEPH to SeaweedFS, especially with lots of images/text/audio files to process.

I found MinIO benchmark results is really, well, "marketing". MinIO is basically just an S3 API layer on top of the local disks. Any object is mapped to at least 2 files on disk, one for metadata and one is the object itself.

SeaweedFS author here. Thanks for your candid answer. You do not need to use multiple SeaweedFS components. Just download the binary and run "weed server -s3".

There are many other components, but you do not really need to use them. This default mode should be good enough for most cases. I saw many times people try to optimize too early, but often unnecessary, and sometimes in the wrong way.

I would like to know what kind of setup you are running. It should beat most other options if the use case needs lots of small files, e.g. millions or billions of files. If just small use case, e.g. a few personal files, it would be an overkill.

Another aspect is how to increase capacity for existing clusters. It should be most simple for SeaweedFS, just start one more volume server. And it will linearly increase the throughput.

Besides storage cost, S3 API access cost can also be high if frequently accessed. And latency is unpredictable.

You can use SeaweedFS Remote Object Store Gateway to cache S3 (or any S3 API compatible vendors) to local servers, and access them at local network speed, and asynchronously sync back to S3.

https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot...

In snowflake id generation mode, the "which volume is writable" information can be read from other follower masters.

This youtuber needs to make a living.

Roblox is great that it built up an ecosystem where people can contribute and get rewarded. It is a positive feedback loop.

Not like open source software, where the financial loop is broken. I am pretty sure the Bolt creator did not get anything from HashCorp for his work.

Your answer should be voted to the top! :)

OSS contributors are rarely noticed or appreciated. Did HashiCorp ever sponsor you or share any revenue with you? The OSS ecosystem is broken.

Thanks for the details! 2 days with 1.5TB vs 3 days with 300GB, that is good to know! :)

btw: If you use "weed filer.copy", it should be much faster. Rsync needs to go through FUSE mount.

This is a fair complaint. :)

For filer metadata, you should just pick the one you are most familiar with.

There is a wiki page for production setup. https://github.com/chrislusf/seaweedfs/wiki/Production-Setup

No preference actually.

SeaweedFS is portable, the data file and the metadata.

Many systems look similar. Ceph, MooseFS, etc. The difference are in details. Not possible to benchmark everything. And each use case has different access pattern, CPU, memory, network. It's better to benchmark with your own use cases.

LeoFS last release is about 3 years ago (v1.4.3 February 20th, 2019), and seems more complicated. SeaweedFS is still growing and has being released on a weekly basis. Just ask if you need any new feature.

Also, think about how to manage. For example, grow capacity. For SeaweedFS, you just need to add one server and point it to the master. That is it!

Actually SeaweedFS does just that, append small objects into larger blocks during write or update, and run garbage collection during compaction.

Aha, I replied above and you are saying the same thing!

Small objects, 4K ~ 4MB, are very common in machine learning area(audio, video, text), or surveillance, game assets, etc. SeaweedFS saw a lot of usage in these areas.

Usually you would benchmark on difficult tasks, not the easiest one. For computers, batch IO operations are much faster than random IO and can easily saturate the network.

This benchmark uses large batch size, 64MB, to test. There is nothing new here. Most common file systems can easily do the same.

The difficult task is to read and write lots of small files. There is a term for it, LOSF. I work on SeaweedFS, https://github.com/chrislusf/seaweedfs , which is designed to handle LOSF. And of course, no problem with large files at all.

SeaweedFS is very different from Nginx. It's just the names are so similar.

There are 2 ways to cache: write through and write back. You are using write through, which needs to write to the remote storage before returning. Write back is only writing to local copy, which is much faster to return. The actual updates are executed asynchronously.

Interesting that SeaweedFS also has a similar named "Gateway to Remote Object Storage" https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot...

The difference is that SeaweedFS can support both read and write, with asynchronous write back. Nginx can support read only caching with 1 hour TTL

Polish cloth max will double the cleaning capability with 2x size, at $29.99, with unprecedented industrial leading price-to-area ratio.

Nice work on qocache!

I checked the code. Is it loading data as "map[string]*list.Element"?

Nice work!

Does Roapi needs to load the data file into memory first, and then answer the queries? Or does it handle the query while streaming through the data?

There are simply too many object store vendors, with different storage pricing, egress pricing, network speed, and reliabilities.

What about using SeaweedFS gateway to remote object store? https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot...

Basically you can think it is a cache. You operate with normal S3 APIs, and the data will be uploaded to remote storage asynchronously (most vendors have free uploading).

It's not a magic pill for all users, but it should be very fast, and cut down the API costs, egress costs.

Most cloud object storage can be a good off-site backup. But accessing your data fast and cheap is not easy.

That is why SeaweedFS added a gateway to remote object store. https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot... , which asynchronously write local changes to the cloud. If there are enough local capacity, there should be no egress cost.

Hopefully, this can change the common pattern, to really treat the cloud object store as a backup.

As a main contributor to an open source project (https://github.com/chrislusf/seaweedfs), I can confirm that this finding is so true.

However, seems this research did not look into Apache projects, which basically maintain a different culture to encourage more contributors, so much as to encourage the main contributors to refrain from jumping to solve an issue until another person steps in first.

The option has 7 languages. So 7 == 80 ?

"And in 2016, Harvard University scientists concluded the molasses disaster would have killed fewer people had it happened in the summer. They figured the winter air cooled and then thickened the goo, making it harder to rescue victims."

Does it take some scientists to figure this out? If so, are there any papers to explain their findings?

I have the same syndrome. But I feel seaweedfs should be easy enough to be picked up. If you have any questions, just let me know.