MinIO is now in maintenance-mode

I think, if anything, in this age of AI coding we should see a resurgence in true open-source projects where people are writing code how they feel like writing it and tossing it out into the world. The quality will be a mixed bag! and that's okay. No warranty expressed or implied. As the quality rises and the cost of AI coding drops - and it will, this phase of $500/mo for Cursor is not going to last - I think we'll see plenty more open source projects that embody the spirit you're talking about.

The trick here is that people may not want to be coding MinIO. It's like... just not that fun of a thing to work on, compared to something more visible, more elevator-pitchy, more sexy. You spend all your spare time donating your labour to a project that... serves files? I the lowly devops bow before you and thank you for your beautiful contribution, but I the person meeting you at a party wonder why you do this in particular with your spare time instead of, well, so many other things.

I've never understood it, but then, that's why I'm not a famous open-source dev, right?

I don't think there's still someone actively working on the Linux kernel without receiving a salary, and this for the last two decades more or less.

Human beings have this strange desire to be fed, have shelter and other such mundane stuff, all of those clearly less important than software in the big scheme of things, of course.

I believe that, at the end of the day, open source enthusiasts still need to make a living.

The beauty of open source is that there are all kinds of reasons for contributing to it, and all are valid. For some, it's just a hobby. For others, like Valve, it's a means of building their own platform. Hardware manufacturers like AMD (and increasingly Nvidia) contribute drivers to the kernel because they want to sell hardware.

Ceph is quite expensive in terms of resource usage, but it is robust and battle-tested. RustFS is very new, very much a work in progress[1], and will probably eat your data.

If you're looking for something that won't eat your data in edge cases, Ceph (and perhaps Garage) are your only options.

[1]https://github.com/rustfs/rustfs/issues/829

Good question. On their website they list 3550 Lenox Road, NE Atlanta, Georgia 30326 as their address. But no info about the company name, CEO or anything like that.

We're in the process of moving to it, and it does seem to have a lot of small bugfixes flying around, but the maintainer is EXTREMELY responsive. I think we'll just end up doing a bit of testing before upgrading to newer versions.

For our use case (3 nodes, 61TB of NVMe) it seems like the best option out of what I looked at (GarageFS, JuiceFS, Ceph). If we had 5+ nodes I'd probably have gone with Ceph though.

Since storage is a critical component, I closely watched it and engaged with the project for about 2 years circa as i contemplated adding it to our project, but the project is still immature from a reliability perspective in my opinion.

No test suite, plenty of regressions, and data loss bugs on core code paths that should have been battled tested after so many years. There are many moving parts, which is both its strength and its weakness as anything can break - and does break. Even Erasure Coding/Decoding has had problems, but a guy from Proton has contributed a lot of fixes in this area lately.

One of the big positive in my opinion, is the maintainer. He is an extremely friendly and responsive gentleman. Seaweedfs is also the most lightweight storage system you can find, and it is extremely easy to set up, and can run on servers with very little hardware resources.

Many people are happy with it, but you'd better be ready to understand their file format to fix corruption issues by hand. As far as i am concerned, i realized that after watching all these bugs, the idea of using seaweedfs was causing me more anxiety than peace of mind. Since we didn't need to store billions of files yet, not even millions, we went with creating a file storage API in ASP.NET Core in 1 or 2 hours, hosted on a VPS, that we can replicate using rsync without problem. Since i made this decision, i have peace of mind and no longer think about my storage system. Simplicity is often better, and OSes have long been optimized to cache and serve files natively.

If you are not interested in contributing fixes and digging into the file format when a problem occurs, and if your data is important to you, unless you operate at the billions of files scalability tier Seaweedfs shines at, i'd suggest rolling your own boring storage system.

Shot you an email about how we can potentially help you with this.

I'm confused why you would want to turn an expensive thing (cloud block storage) into a cheaper thing (cloud object storage) with worse durability in a way that is more effort to run?

I'm not saying it's wrong since I don't know what it's for, I'm just wondering what the use-case could be.

Hetzner VM with mounted storage box.

https://www.hetzner.com/storage/storage-box/

It's not as fast as local storage of course, but it's cheap!

Congratulations on earning that opportunity!

Thank you for your work. I was in a position where I had to choose between minio and seaweed FS and though seaweed FS was better in every way the lack of an includes dashboard or UI accessibility was a huge factor for me back then. I don't expect or even want you to make any roadmap changes but just wanted to let you know of a possible pain point.

MinIO had a de facto CLA. MinIO required contributors to license their code to the project maintainers (only) under Apache 2. Not as bad as copyright assignment, but still asymmetric (they can relicense for commercial use, but you only get AGPL). https://github.com/minio/minio/blob/master/.github/PULL_REQU...

The FSF can't pull the rug because of its bylaws

Well, maybe they are using that experience to build something better this time around? One can hope...

I use garage at home, single node setup. It's very easy and fast, I'm happy with it. You're missing out on a UI for it, but MountainDuck / CyberDuck solves that problem for me.

Yeah, that page is horrendous and looks super sketchy. It looks like a very professional fishing attempt to get unsuspecting developers to download malware.

They have a lot of obviously fake quotes from non-existent people at positions that don’t even mention what company it is. The pictures are misgendered and even contain pictures of kids.

Feels like the whole page is AI generated.

I saw an article here not long about where someone explained they were hosting their Kopia or Nextcloud aver Garage, but I can't find it anymore.

This was going to be my next project, as I am currently storing my Kopia/Ente on MinIO in a non-distributed way. MinIO project going to shi*s is a good reason to take on this project faster than later.

We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s)

I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.

Speaking as an open-source enthusiast, I’m actually really digging RustFS. Honestly, anything that can replace or compete with MinIO is a win for the users. Their marketing vibe feels pretty American, actually—they aren't afraid to be loud and proud, haha. You gotta give it to them though, they’ve got guts, and their timing is spot on.

> I maintain an S3 client that has a test matrix for the commonly used S3 implementations.

Is it open to the public? I'd like to check it out

Because they aren't thinking about all the chinese wetware they'd be writing down that decision with.

Probably just me but I would stay away from anything saying vibecoder in the repo

I don't see a contributor licensing agreement (CLA), so you may be right.

(I personally choose not to contribute to projects with CLAs, I don't want my contributions to become closed-source in the future.)

I don't think there would be an issue with removing AGPL contributed code. You can't force someone to distribute something they don't want to. IANAL, but I believe that what (all?) copyright in software is most concerned with is the active distribution of code -- not the removal of code.

That said, if there was contributed AGPL code, they couldn't change the license on that part of the code w/o a CLA. AGPL also doesn't necessarily mean you have to make the code publicly available, just available to those that you give the program to (I'm assuming AGPL is like the GPL in this regard).

So, that I'd be curious about it is -- (1) is there any contributed AGPL code in the current version? (2) what license is granted to customers of the enterprise version?

Minio can completely use whatever license they want for their code. But, if there was contributed code w/o a CLA, then I'm not sure how a commercial/enterprise license would play with contriubuted AGPL code. It would be an interesting question to find out.

I'm not a contributor to Minio. This is its own separate thing.

I do have a separate AGPL project (see github) where I have nearly all of the copyright and have looked into how one would be able to enforce this in the US at some point and it did look pretty bleak -- it is a civil suit where you have to show damages etc. but IANAL.

I did not like the FUD they were spreading about AGPL at the time since it is a good license for end-user applications.

MinIO had a de facto CLA. MinIO required contributors to license their code to the project maintainers (only) under Apache 2. Not as bad as copyright assignment, but still asymmetric (they can relicense for commercial use, but you only get AGPL). https://github.com/minio/minio/blob/master/.github/PULL_REQU...

It uses LMDB, so if the object mapping fits in memory that should be pretty optimal for reading, while using the build-in Linux page cache and not a separate one (important for testing use cases). For write/deletes it has a bit of write-amplification due to the copy-on-write btree. I've implemented a separate, optional WAL for this and also a mode where writes/delete can be bundeled in a transaction, but in practice I think the performance difference should not matter.

W.r.t. filesystem: Yes, aware of this. Initially used minio and also implemented the use case directly on XFS as well and only had problems at larger scales (that still fit on a machine) with it. Ceph went into a similar direction with BlueStore (reimplementing the filesystem, but with RocksDB).

Oh no, I used MinIO once or twice for some unlicensed software.

Should I contact a MinIO salesman to purchase an enterprise license ASAP or is it fine if I license my kids and advent of code solutions under the AGPLv3 license ?

  > you may be violating AGPLv3 if you are using MinIO to build commercial products or using it for commercial purposes

Yeah, this is bullshit. I wish the guy used his own advise and spoke to a lawyer :)

Wait, what's the consensus on this? Are they saying that using object storage over a standard network API which they didn't even create, makes your application a derivative work of the object store?

Or just that the users would need to make minio sources, including modifications, freely available?

I guess that's kind of the big question inherent to the AGPL?

yes

Well I didn't mind when they removed it and certainly I didn't consider their paid version which is way too expensive for most use cases.

The UI was useful when first configuring the buckets and permissions; if you've got it working (and don't need to change anything) you're good to go. Also, everything can be configured without the UI (not so easily of course).

Yes this is concerning for me too. Hopefully if they don't fix/merge security issues somebody will fork and maintain it. It shouldn't be too much work. I'd even do it myself if I was experienced in golang.

I think the criticism (just interpreting the post, don’t know anything about the technical situation) is that the complication is not necessary/worthwhile.

POSIX can be complicated, but it puts you in a nice ecosystem, so for some use-cases complex POSIX support is not over complicated. It is just… appropriately complicated.

What would go in to POSIX compatibility for a product like this which would make it complicated? Because the kind of stuff that stands out to me is the use of Linux specific syscalls like epoll/io_uring vs trad POSIX stuff like poll. That doesn't seem too complicated?

So when we say "they abandoned posix compatibility", are we saying "They abandoned the POSIX filesystem storage backend"? I believe that's true, I used to use minio on a FreeBSD server but after an update I had to switch to just passing in zfs block devs.

Or are we saying that they no longer support running minio on POSIX systems at all, due to using linux specific syscalls or something else I'm not thinking of? I don't know whether they did this or not.

Those seem like two very different things to me, and when someone says "they don't support POSIX", I assume the latter

I've done some preliminary testing with garage and I was pleasantly surprised. It worked as expected and didn't run into any gotchas.

This is the winner

I just learned about the rclone serve subcommand the other day. Rclone is not exactly niche, but it feels like such an underrated piece of software.

Versity is really promising. I got a chance to meet with Ben recently at the Super Computing conference in St. Louis and he was super chill about stuff. Big shout out to him.

He also mentioned that the minio-to-versity migration is a straight forward process. Apparently, you just read the data from mino's shadow filesystem and set it as an extended attribute in your file.

I really like what I've (just now) read about Versity. I like that they are thinking about large scale deployments with tape as the explicit cold-storage option. It really makes sense to me coming from an HPC background.

Thanks for posting this, as it's the first I've come across their work.

Garage also decide to not implement erasure coding.

Might be because of this other storage product named that https://github.com/NVIDIA/aistore

It can store things for AI workloads (and non-AI workloads, but who’s counting…)

Looks like AI slop

    Replication

    A trusted identity provider is a
    key component to single sign on.

Uh, what?

It’s probably just Minio but it costs more money.

About a billion dollars difference in valuation up until the bubble pops.

back in 2018, it didn't feel this way

They required a "Community Contribution License" in each PR description, which licensed each contribution under Apache 2 as an inbound license.

Meanwhile, MinIO's own contributions and the distribution itself (outbound license) were AGPL licensed.

It's effectively a CLA, just a bit weaker, since they're still bound by the terms of Apache 2 vs. a full license assignment like most CLAs.

What if a company change license, drop the project or goes bankrupt?

You shouldn't expect guarantee of any kind.

You can, sort of, sometimes. Copyleft is still based on copyright. So in theory you can do a new license as long as all the copyright holders agree to the change. Take open source/free/copyleft out of it:

You create a proprietary piece of software. You license it to Google and negotiate terms. You then negotiate different terms with Microsoft. Nothing so far prevents you from doing this. You can't yank the license from Google unless your contract allows that, but maybe it does. You can in theory then go and release it under a different license to the public. If that license is perpetual and non-revokable then presumably I can use it after you decide to stop offering that license. But if the license is non-transferrable then I can't pass on your software to someone else either by giving them a flash drive with it, or by releasing it under a different license.

Several open source projects have been re-licensed. The main thing that really is the obstacle is that in a popular open source or copyleft project you have many contributors each of which holds the copyright to their patches. So now you have a mess of trying to relicense only some parts of your codebase and replace others for the people resisting the change or those you can't reach. It's a messy process. For example, check out how the Open Street Maps data got relicensed and what that took.

It is also useful to remember that MinIO has historically held to an absurd interpretation of the AGPL -- that it spreads (again, according to them) to software that communicates with MinIO via the REST API/CLI.

I assume forks, and software that uses them will be held to the same requirements.

So they’re pulling an OpenAI.

Start open source to use free advertising and community programmer, and then dumps it all for commercial licensing.

I think n8n is next because they finished the release candidate for version 2.0, but there are no changelogs.

Everything uses poorly documented, sometimes inconsistent HTTP headers that read like afterthoughts/tech debt. An S3 standard implementation has to have amazon branding all over it (x-amz) which is gross.

Last I checked the user guide to the API was 3500 pages.

3500 pages to describe upload and download, basically. That is pretty strange in my book.

It gets complex with ACLs for permissions, lifecycle controls, header controls and a bunch of other features that are needed on S3 scale but not at smaller provider scale.

And many S3-compatible alternatives (probably most but the big ones like Ceph) don't implement all of the features.

For example for lifecycles backblaze have completely different JSON syntax

That may be what S3 is like, but what the S3 API is is this: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3

My browser prints that out to 413 pages with a naive print preview. You can squeeze it to 350 pretty reasonably with a bit of scaling before it starts getting to awfully small type on the page.

Yes, there's a simple API with simple capabilities struggling to get out there, but pointing that out is merely the first step on the thousand-mile journey of determining what, exactly, that is. "Everybody uses 10% of Microsoft Word, the problem is, they all use a different 10%", basically. If you sat down with even 5 relevant stakeholders and tried to define that "simple API" you'd be shocked what you discover and how badly Hyrum's Law will bite you even at that scale.

!!!

I’ve seen a lot of bad takes and this is one of them.

Listing keys is weird (is it V1 or V2)?

The authentication relies on an obtuse and idiosyncratic signature algorithm.

And S3 in practice responds with malformed XML, as you point out.

Protocol-wise, I have trouble liking it over WebDAV. And that's depressing.

HTTP isn't really a great back plane for object storage.

Like everything it starts off simple but slowly with every feature added over 19 years Simple Storage is it not.

S3 has 3 independent permissions mechanisms.

I suspect there is no decent "minimal" API. Once you get to tens of millions of objects in a given prefix, you need server side filtering logic. And to make it worse, you need multiple ways to do that.

For example, did you know that date filtering in S3 is based on string prefix matching against an ISO8601/RFC3339 style string representation? Want all objects created between 2024-01-01 and 2024-06-30? You'll need to construct six YYYY-MM prefixes (one per month) for datetime and add them as filter array elements.

As a result the service abbreviation is also incorrect these days. Originally the first S stood for "Simple". With all the additions they've had to bolt on, S2 would be far more appropriate a name.

MinIO still makes no sense, as Ceph is fundamentally already RADOS at its core (fully compatible with S3 API.)

For a proper Ceph setup, even the 45drives budget configuration is still not "hobby" grade.

I will have to dive into the MinIO manual at some point, as the value proposition still seems like a mystery. Cheers =3

Consider migrating to VictoriaMetrics and saving on storage costs and operations costs. You also won't need MinIO, since it stores data to local filesystem (aka to regular persistent volumes). See real-world reports from happy users who saved costs on a large-scale Prometheus-compatible monitoring - https://docs.victoriametrics.com/victoriametrics/casestudies...

But what's the point to use these DIY object storage systems, when they do not provide durability and other important guarantees provided by S3 and GCS?

MinIO also allows losing your data, since it doesn't provide high durability guarantees unlike S3 and GCS.

> What is the use case for implementing a POSIX filesystem on top of an object store?

The use case is fully stateless infrastructure: your file/database servers become disposable and interchangeable (no "pets"), because all state lives in S3. This dramatically simplifies operations, scaling, and disaster recovery, and it's cheap since S3 (or at least, S3 compatible services) storage costs are very low.

The MinIO article's criticisms don't really apply here because ZeroFS doesn't store files 1:1 to S3. It uses an LSM-tree database backed by S3, which allows it to implement proper POSIX semantics with actual performance.

> OS core and paid hosting with bells and whistles has proven to be a good sustainable model

Redis, Elasticsearch, Terraform, MongoDB, CockroachDB have all changed their OSS licenses in recent years.

I've been looking at microceph, but the requirement to run 3 OSDs on loopback files plus this comment from the docs gives me pause:

`Be wary that an OSD, whether based on a physical device or a file, is resource intensive.`

Can anyone quantify "resource intensive" here? Is it "takes an entire Raspberry Pi to run the minimum set" or is it "takes 4 cores per OSD"?

Edit: This is the specific doc page https://canonical-microceph.readthedocs-hosted.com/stable/ho...

Ceph is awesome for software defined storage where you have multiple storage nodes and multiple storage devices on each. It's way too heavy and resource intensive for a single machine with loopback devices.

ceph depends a lot on your use case

minio was also suited for some smaller use cases (e.g. running a partial S3 compatible storage for integration tests). Ceph isn't really good for it.

But if you ran large minio clusters in production ceph might be a very good alternative.

If you just need a s3 endpoint for some services lookup garage

Objects in Garage are broken up into 1MB (default) blocks, and compressed with zstandard. So, it would be difficult to reconstruct the files. I don't know if that was a recent change since you looked at it.

Configuration is still through the CLI, though it's fairly simple. If your usecase is similar to the way that the Deuxfleurs organization uses it -- several heterogeneous, geographically distributed nodes that are more or less set-it-and-forget-it -- then it's probably a good fit.

"For enterprise support and actively maintained versions, please see MinIO AIStor."

Commercial only, they will replace the agpl contributions from external people. (Or at least they will say that)

this is no different than grok vs groq imo. aistor and aistore are different names, even if they're pronounced similarly.