(I work on SeaweedFS) Needs to pay to join CNCF, about $8k/year. I would like more sponsorship.
203 karma · joined October 20, 2011
AWS_ACCESS_KEY_ID=admin AWS_SECRET_ACCESS_KEY=secret S3_BUCKET=my-bucket ./weed mini -dir=./data
That's it.
The S3 endpoint is at http://localhost:8333, my-bucket exists, and admin/secret are valid credentials.
It also starts an Iceberg REST catalog and S3 Table Buckets for Iceberg and Lance tables, with table maintenance jobs.
Actually, AI does help a lot to setup all variations of testing scenarios and uncover edge case bugs. It's more helpful than vibes.
And you do need to be systematic to examine AI generated code. There are no shortcut for it.
(I work on SeaweedFS)
(I work on SeaweedFS)
Just download the single binary, for most platforms, and run "weed mini -dir=your_data_directory", with all the configuration optimized.
I am trying to support AWS S3 APIs as complete as possible.
Recently added support for Table Bucket, besides myriads of details, such as policies, STS, IAM, OIDC, WORM, lock and versioning, governance, etc.
Haha, you used Claude to find the Clause code.
I used Claude to generate a lot of admin UI pages, saved a lot of time. The core storage engine part I dare not using AI, same as you.
SeaweedFS was started as a learning project and evolves along the way, getting ideas from papers for Facebook Haystack, Google Colossus, Facebook Tectonics. With its distributed append-only storage, it naturally fits object store. Sorry to see MinIO went away. SeaweedFS learned a lot from it. Some S3 interface code was copied from MinIO when it was still Apache 2.0 License. AWS S3 APIs are fairly complicated. I am trying to replicate as much as possible.
Some recent developments:
* Run "weed mini -dir=xxx", it will just work. Nothing else to setup.
* Added Table Bucket and Iceberg Catalog.
* Added admin UI
Why skipping SeaweedFS? It rank #1 on all benchmarks, and has a lot of features.
Just run "weed sever -s3 -dir=..." to have an object store.
Recently SeaweedFS is moving fast and added a lot more features, such as: * Server Side Encryption: SSE-S3, SSE-KMS, SSE-C * Object Versioning * Object Lock & Retention * IAM integration * a lot of integration tests
Also, SeaweedFS performance is the best in almost all categories in a user's test https://www.repoflow.io/blog/benchmarking-self-hosted-s3-com... And after that, there is a recent architectural change that increases performance even more, with write latency reduced by 30%.
Sorry, everybody has different background of knowledge. Hard to understand where the question comes from. I think https://www.usenix.org/system/files/fast21-pan.pdf may be helpful here.
A blob has its own storage, which can be replicated to other hosts in case current host is not available. It can scale up independently of the file metadata.
Files and S3 are higher layers above the blob storage. They require metadata to manage to the blobs, and other metadata for directories, S3 access, etc.
These metadata usually sit together with the disks containing the files. But in highly scalable systems, the metadata has dedicated stores, e.g., Google's Colossus, Facebook's Techtonics, etc. SeaweedFS file system layer is built as a web application of managing the metadata of blobs.
Actually SeaweedFS file system implementation is just one way to manage the metadata. There are other possible variations, depending on requirements.
There are a couple of slides on the SeaweedFS github README page. You may get more details there.
The allocated storage is append only. For updates, just allocate another blob. The deleted blobs would be garbage collected later. So it is not really mmap.
> Also what is the difference between a file, an object, a blob, a filesystem and an object store?
The answer would be too long to fit here. Maybe chatgpt can help. :)
> Is all this just files indexed with sql?
Sort of yes.