Hacker News Reader

Comment by wvh | Hacker News Reader

Apart from Minio, we tried Garage and Ceph. I think there's definitely a need for something that interfaces using S3 API but is just a simple file system underneath, for local, testing and small scale deployments. Not sure that exists? Of course a lot of stuff is being bolted onto S3 and it's not as simple as it initially claimed to be.

View on HN

SeaweedFS's new `weed mini` command[0] does a great job at that. Previously our most flakey tests in CI were due to MinIO sometimes not starting up properly, but with `weed mini` that was completely resolved.

[0]: https://github.com/seaweedfs/seaweedfs/wiki/Quick-Start-with...

Minio started like that but they migrated away from it. It's just hard to keep it up once you start implementing advanced S3 features (versioning/legal hold, metadata etc.) and storage features (replication/erasure coding)

There's plenty of middle ground between "just expose underlying FS as objects, can't support many S3 operations well" and "it's a single node not a distributed storage system".

For my personal use, asynchronous remote backup is plenty for durability, a Bee-Link ME Mini with 6x4TB NVMe is plenty of space for any bucket I care to have, and I'd love to have an S3 server that doesn't even attempt to be distributed.

> for local, testing and small scale deployments

Yes I'm looking for exactly that and unfortunately haven't found a solution.

Tried garage, but they require running a proxy for CORS, which makes signed browser uploads a practical impossibility for the development machine. I had no idea that such a simple popular scenario is in fact too exotic.

What about s3 stored in SQLite? https://github.com/seddonm1/s3ite

This was written to store many thousands of images for machine learning

From what I can gather, S3Proxy[1] can do this, but relies on a Java library that's no longer maintained[2], so not really much better.

I too think it would be great with a simple project that can serve S3 from filesystem, for local deployments that doesn't need balls to the walls performance.

[1]: https://github.com/gaul/s3proxy

[2]: https://jclouds.apache.org/

The problem with that approach is that S3 object names are not compatible with POSIX file names. They can contain characters that are not valid on a filesystem, or have special meaning (like "/")

A simple litmus test I like to do with S3 storages is to create two objects, one called "foo" and one called "foo/bar". If the S3 uses a filesystem as backend, only the first of those can be created

I've been looking into rclone which can serve s3 in a basic way https://rclone.org/commands/rclone_serve_s3/

For testing, consider https://github.com/localstack/localstack

WAY too much. I just need a tiny service that translates common S3 ops into filesystem ops and back.

Try versitygw.

I’m considering it for a production deployment, too. There’s much to be said for a system that doesn’t lock your data in to its custom storage format.

Reply on news.ycombinator.com