Hacker News Reader

Comment by 0x000000001 | Hacker News Reader

Be warned that their code quality is pretty bad. There was a bug I was dealing with last year where it did not delete objects but returned the correct HTTP response code indicating it did. This was widespread, not just some edge case I encountered. Their broken test suite doesn't actually verify the object on disk changed. I tried to engage them but they blew me off.

View on HN

Minio isn't durable. Any S3 operation might not be on disk after it is completed successfully. They had an environment variable MINIO_DRIVE_SYNC for a while that fixed some cases. Looking at the current code this setting is called MINIO_FS_OSYNC now (for some reason) https://github.com/minio/minio/pull/9581/commits/ce63c75575a... (but I wouldn't trust that... are they fsyncing directories correctly? Making sure object metadata gets deleted with the data in one transaction etc.). Totally undocumented, too.

I guess this makes minio "fast". But it might eat your data. Please use something like Ceph+RadosGW instead. It might be okay for running tests where durability isn't a requirement.

That had me curious, so I searched a bit in their issues.

Their attitude about it isn't great: https://github.com/minio/minio/issues/3536

That's too bad, as it seems well thought out in other areas, like clustering.

MinIO team care about an issue if you are paid customer, not for people who use the open source. Indeed MinIO is not even fully S3 compatible with many edge cases and close the issues related to it by saying it’s not a priority.

You might want to look at other options as well like SeaweedFS [0] a POSIX compliant S3 compatible distributed file system.

[0] https://github.com/chrislusf/seaweedfs

I haven't used seaweedfs yet, but it looks better (and small file/object performance should be miles better). W.r.t. to fsync/durability with the seaweedfs API to a volume server you have to turn fsync on via paramter and it is disabled by default. With S3 it is probably also off by default and you can turn it on per bucket: https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Co... .

Both should default to fsync on, with the option to turn it off. So not a great choice of defaults. Again, it probably looks good in benchmarks when people naively compare S3 stores. But it just shouldn't eat your data per default.

We tried to use it a year ago or so, because of the performance promise. We were getting random glitches every few thousand files during operations. There was no obvious pattern, so difficult to reproduce, and as far as I remember there was mentions of it in github. Hopefully they acknowledge and get over this hump, as it seems like a promising project all together.

Yep, same experience unfortunately

I also hit a very frustrating issue in minio where CORS headers weren't being set properly and there were many similar cases in their issues history. Their response was basically "works for me, sorry".

I'm pretty sure there was something weird going on with how minio was reading the config state, as I definitely was not the only one hitting it. Luckily I only had to use it for local testing in the project, but the whole thing didn't leave me feeling good.

[1] https://github.com/minio/minio/issues/11111

Github issue link? They seem to have a solid ci setup, and I know several large enterprises using. But I found a bug for my usage != bad code quality.

My usage was "setup a basic single node for testing, upload a file with mc client, delete a file with mc client". They failed that test. It was responding with 200s but the file was never deleted.

There are loads of issues like this on their github: https://github.com/minio/minio/issues/8873

That's an interesting issue. Boiled down to the object name having '//' in it, which drove a certain direction for the shard location that wasn't the same shard location that the delete function looked in.

Sounds like the shard hashing happens before or after object name normalization depending on the operation. Ouch.

Based on sib comment, it looks like it was related to a poor name sanitation / matching function (which says, to me, that's risky if you have untrusted names), but this could also be caused by a lazy or delayed deletion strategy.

I had issues with frequent crashes due to various panics a while a go. It eventually went away after a version upgrade. But now reading this I don’t feel terribly confident in using minio long term.

Is Ceph with Rados Gateway a better alternative to this?

I have a 500PB Ceph setup @ work, but I don't maintain it. It's been solid.

I would say no in production. I was recently testing a ceph + rgw as an on prem s3 solution, but high throughput puts + ls caused an index corruption that “lost” files according to future LS’s, the file was still there if you directly get it. When this was reported it was already found multiple years ago, and never fixed

Could you reference a bug url? I tried to find it via tracker.ceph.com but failed to do so (I don't claim that the problem doesn't exist). That said referencing a bug url would be nice if you want to increase credibility of your claim.

Could be: https://tracker.ceph.com/issues/24744

I know this bug has hampered our use of ceph at singlestore. Note that this is not an eventual consistency issue. When it happens the list command will permanently miss files.

CERN runs at least in part on Ceph and it's well documented:

https://www.youtube.com/watch?v=OopRMUYiY5E

100% agree. I pin the version we use because you never know if it will come with even more bugs.

what would be a better way to export a nfs storage to s3 than? swift, like it does for glusterfs?

Reply on news.ycombinator.com