Comment by tobias3 | Hacker News Reader

Comment by tobias3 | Hacker News Reader

Parent Full thread

Minio isn't durable. Any S3 operation might not be on disk after it is completed successfully. They had an environment variable MINIO_DRIVE_SYNC for a while that fixed some cases. Looking at the current code this setting is called MINIO_FS_OSYNC now (for some reason) https://github.com/minio/minio/pull/9581/commits/ce63c75575a... (but I wouldn't trust that... are they fsyncing directories correctly? Making sure object metadata gets deleted with the data in one transaction etc.). Totally undocumented, too.

I guess this makes minio "fast". But it might eat your data. Please use something like Ceph+RadosGW instead. It might be okay for running tests where durability isn't a requirement.

That had me curious, so I searched a bit in their issues.

Their attitude about it isn't great: https://github.com/minio/minio/issues/3536

That's too bad, as it seems well thought out in other areas, like clustering.

MinIO team care about an issue if you are paid customer, not for people who use the open source. Indeed MinIO is not even fully S3 compatible with many edge cases and close the issues related to it by saying it’s not a priority.

You might want to look at other options as well like SeaweedFS [0] a POSIX compliant S3 compatible distributed file system.

[0] https://github.com/chrislusf/seaweedfs

I haven't used seaweedfs yet, but it looks better (and small file/object performance should be miles better). W.r.t. to fsync/durability with the seaweedfs API to a volume server you have to turn fsync on via paramter and it is disabled by default. With S3 it is probably also off by default and you can turn it on per bucket: https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Co... .

Both should default to fsync on, with the option to turn it off. So not a great choice of defaults. Again, it probably looks good in benchmarks when people naively compare S3 stores. But it just shouldn't eat your data per default.

Reply on news.ycombinator.com