Hacker News Reader

Comment by Lunatic666 | Hacker News Reader

Is there any open source S3 compatible software? I know Riak Cloud Storage (http://docs.basho.com/riak/cs/2.1.1/), but I think it’s not maintained anymore.

View on HN

Minio and Ceph both fit this description, with the latter also offering block and file access. Swift is close, but a slightly different native API (there's an adapter but I'm not sure of its status). Scality has open-sourced their implementation as Zenko. Also LeoFS (which despite the name is not a filesystem).

https://minio.io/

I believe minio.io provides open source S3 compatible storage on your own servers.

True

There is pithos [1] from the Exoscale [2] and powers the production system. A new version of the open source project with API signature v4 and other updates is due out soon.

[1] http://pithos.io/ [2] https://www.exoscale.com/object-storage/

Joyent's Manta has an S3 adapter, all open-source.

OpenStack Swift or Ceph with Ceph Object Gateway. Don't use minio, it's a toy for testing.

I've never used it in production; in what ways is minio lacking compared to Swift and Ceph?

S3 is an abstraction that gives you a limited amount of operations with limited guarantees (e.g. eventual consistency, can only replace whole files). What you get in return is easier scalability and performance. If you use an S3 API to store files (like minio does) you give up power and gain nothing. So you are better off using NFS, samba, webdav, ftp, etc. Additionally minio doesn't seem to sync files to the file system, so you can't be sure a file is actually stored after a PUT operation (AWS S3 and swift have eventual consistency and Ceph has stronger guarantees).

That's curious, because the OpenStack Swift docs say "Objects are stored as binary files on the filesystem with metadata stored in the file’s extended attributes (xattrs)".

As far as I know, the advantage comes from exposing a course-grained API over the network, which is generally more efficient and reliable than doing many small operations. It would be hard to implement Minio's distributed mode using the filesystem API.

What's wrong with minio? It does its job and does it well. I did backups with it once upon a time, in production, and it never failed.

That's libelously untrue. You might think Ceph or Swift are better, that's fine, but it's no toy. I've seen a few toy S3 implementations. I even called out Zettar on my blog back in the day. If it's not scalable or highly available it's a toy. If it's unstable it's a toy. If it's super-duper slow it's a toy. Minio is none of these things, plus it has features like erasure coding and encryption that are mature enough to be backed by real support. Best ever? I'm not saying that, but it's no toy.

Disclaimer: I have no affiliation with Minio. In fact, they're very nearly a competitor with the project I work on. However, Minio is full of ex-Gluster people and I consider many of them friends.

The rclone project supports S3

https://rclone.org/s3/

Rclone.org

Reply on news.ycombinator.com