Elasticsearch is open source, again
elastic.co
elastic.co
If you want to work with large companies, being SOC certified makes it easier. Part of that is ensuring your vendors are also compliant with good standards and that's best done with SOC reports.
Although few companies have minimum ticket size for enterprise clients and that is a bad thing IMO.
We did not get certified because we wanted it, we did because the enterprise scale customers forced us to. Due to their internal bureaucracy.
https://news.ycombinator.com/item?id=41304228
This SSO cost post is also interesting:
Identity management, role based access control, useful audit logs; all "enterprise" features, probably are very expensive to implement, and make for obvious "up-charge" product segmentation.
I suspect there's some combination of "the community doesn't add useful implementations of these features" and "we can't possibly risk our reputation based on some community contribution" and "we can use this to segment our product to sell to some and give it away to some."
This set of features seems to always get put in the "enterprise, only for licensed / supported customers" and it stinks.... I can understand why, though, and none of these are strictly speaking "security" as much as "compliance"
The original license owner, if a commercial enterprise trying to sell the product alongside the "open" version, has less incentive to accept those features from the community as it would reduce their sales of the enterprise version of the same thing, and may not align with their long-term product roadmap.
In open source, the team managing a codebase isn't under any obligation to accept contributions the community and you are welcome to fork the project, if you like.
Let's say you work at a company that uses Elasticsearch. Let's say you're running a newspaper and you've got your logs in elasticsearch. Let's say one of your reporters ends up getting chopped up while they're visiting the Ostrich embassy to get a marriage license. Now let's say you're then asked "who looked at the logs of the CMS who searched for and found the IP address that was used by that reporter on October 1st 2018"
That example, purely hypothetical, is an example of "security" but not the typical security you'll see in some open source application -- it's an enterprise "compliance" feature that won't be trivial to implement and will be judged not just on completeness but on user interface, ease of use, ease of implementation, etc.
"security" means different things to different people
I have taken this position in another thread a while ago, but the responses seemed to indicate that this is not a clearly cut situation at all. If it was, what is the point of the "source-available" licenses in the first place? I mean, the idea that they were invented to cut out AWS is pretty prevalent, no?
FSF/GNU have an example of an AGPL proxy becoming compliant by serving it a page with the offer to download source code on the first request, pretty far off from reality if you ask me. That's also the big other issue, AGPL is unclear about conveyance over a network. Does a header work? Does a link to the source repo work or do you need to offer hard copies? What do you do if the "networking" is a highly specific protocol that simply can't make that offer over the wire?
I much prefer the clarity of intent of the EUPL.
Not sure what their counsel is thinking there..
They seem to have at least fixed their compliance page. It used to read:
"If you are an Original Equipment Manufacturers, a Reseller, or an Independent Software Vendor that combines and distributes commercially licensed software with MinIO software and do not wish to distribute the source code for the commercially licensed software under GNU Affero General Public License, Version 3.0 (AGPLv3), you must enter into a commercial license agreement with MinIO, available at https://min.io/pricing."
And that's opposed to "FOSS".
https://web.archive.org/web/20210415185046/https://min.io/co...
Specifically, if you offer the software for "Remote Network Interaction" (AGPLv3 section 13), well, "if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version".
I think the original challenge with AGPLv3 vs (to grossly generalize) the VC-backed open source corporate ecosystem was not around source code, but around monetization as SaaS by the hyperscalers. The problem there is even if the hyperscalers publish source code modifications (which they probably have no problem with) they have such sales efficiency and gravitational pull that they will end up eating your business.
It's not a "can't" it's a "won't".
I suppose that they might not have accepted something that was too small percent wise and hence might have preferred to go head to head no matter where that might have gone.
My real sense for why they've struggled to out maneuver is their lack of execution on their managed service (9 years in market, still minority of their revenue); while you had a head start and I'm sure that's what they point to as preventing execution, if they had really focused there they might be more like Confluent in terms of being considered the well regarded SaaS leader in their segment.
But I do think it'd be a good look for AWS to proactively help these companies. I didn't think the approach taken with Grafana Labs was right... that looked more like a Faustian bargain to an outside observer (e.g. we'll cut you down at your knees and directly compete but offer you their more expensive version on our paper. It looked incredibly humiliating).
But how much value does "a good look" have to AWS?
High vendor management overhead is a huge pain for smaller companies that don't have robust IT to manage those relationships.
The smaller/mid size startups I've worked at almost never acquire "enterprise" software and always leverage pay-by-credit-card type SaaS
Besides operational overhead there's also a much longer acquisition time (no one wants to spend 3-6 months working with a sales team to sign a contract on a project with 2-3 month timeline)
But they made it very difficult to try it out at scale (we generate quite a lot of logs) and at one point they only wanted to talk to the CTO instead of the persons in charge of the PoCs.
That move made them untrustable to me, and they were disqualified from the process. If they wanted to compete on selling the solution to non-technical people that told us all we needed to know about them and how support would be. We ended up choosing managed Opensearch by AWS, which was a shitshow in several fronts. I wish we had given Loki a bigger chance at that time. We've ended up migrating to it anyway.
I was also assisting in the deal with Grafana, which I thought was a good deal on both sides, setting up a framework for AWS and Grafana to work together over a longer timeframe. Ash Manzani who negotiated the deal for AWS later joined Grafana to run Corp Dev for them.
The answer is no, because Linux is open source. It is a multi stakeholder model where no single actor is allowed to control other actors use of the product. There is ownership, but it is separated from control. This is implemented with the GPL, but the license is only a tool to achieve the outcome, a multi stakeholder product.
This is why Amazon, Red Hat and Google all can justify to employ hundreds of engineers all contributing to a common product. Amazon can work on security functionality with no risk that Red Hat will veto it because it threatens its revenue stream. And while none of the top kernel developers have made billions from their important work, they still earn well, and all the mentioned companies have grown to become billion dollar companies.
Everyone knew this in the 90s, that's why we have the philosophy around open source. Now the discourse has changed. It is suddenly immoral to earn money from someone else's product, because if you start a project then you own it outright. Not only that, but you also have a right to become rich from your work. Discussions how immoral it is for a large company to use a piece of software without paying the original author is a completely normal thing to do, never mind that they would have no problem reinventing that particular wheel in a heartbeat.
Companies like Elastic have latched on to this, and call their product open source, but call foul when other people build products and make a living from their software. They're not actually interested in a multi stakeholder model at all.
How big would Wordpress have been without every cloud actor out there offering to host instances for a cheap fee?
This can't be stressed enough. OSS does monetarily work for the developers too. FAANGs love paying top dollar for OSS maintainers.
Personally, I do wish that there was more broad acceptance of the Elastic License. Who wants to put in years building a business and then have a competitor with better distribution take your code and compete directly with you? For me, the reasons to want open-source code are:
* If a vendor goes under, I can self-host
* If a vendor raises prices too much, I can self-host
* If there's a bug in the code that affects me too much, I can fix it
* If there's a feature I really need, I can add it
The Elastic License allows for all of the above. Seems fair to me.
And of course the third party can compete on price, they don't have to develop the actual software they're selling!
Without competition, they are free to charge any rent they like for it.
If you think that the person that originally wrote Redis or Elastic should have an exclusive license to charge people to use that software, that's a totally valid opinion and a totally valid licensing/business model. However, it has nothing to do with open source software.
BSL and GPL code are probably never mixing since they prohibit each other. This creates friction in GPL world and tends to produce incidents line this [1] out of thin air.
It's worth mentioning that this is true -- to an extent. Under ELv2, if the vendor goes under, you can self-host, but you will eventually lose access to any features protected by a license key if/when that license expires, since said vendor can no longer renew said license.
This was one of the main drivers for me writing the FCL [0], which undergoes DOSP [1], even for the protected features.
[0]: https://fcl.dev
At least with DOSP, eventually the community will be able to do those things.
Specifically, this is the text [0]:
> if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version
There are a few companies who try to make it sound like if you interact with an AGPL program over a network then your client code is now infected with the AGPL, but I'm not at all sure how they arrived at that conclusion unless it was willful misinterpretation.
Under the mainstream view, you only have to publish the source code for the AGPL work that you modified, which for 99.9% of users is fine but isn't great for a reseller.
The main barrier isn't the actual text of the license, it's that AGPL is still untested in court and there are companies who will try to make it mean something different than its apparent meaning, so legal departments are liable to get antsy. But lawyers are likely to get antsy about self-hosting under these other licenses as well.
If the AGPL is exactly as you say, I don’t see why this would be a problem for a re-seller. For a pure re-seller I don’t think the value add is provided by modifying the software.
E.g. take the example that Amazon hosts the service and integrates with their internal services etc for logging, storage, load balancing etc. If they only have to distribute the modified source, then their internal service APIs will be leaked. This is probably fine most of the time, but what if the API reveals too much of the secret sauce (very unlikely, but possible so necessary for legal CYA, or extra approvals every time you want to modify the AGPL code). In a more devil’s advocate reading, the following stands out (IANAL, just conjecturing):
> all the source code needed to generate, install, and (for an executable work) run the object code
Do I need to include my super secret storage engine X because my modification requires that I use it? Let’s say I write the code in a way that it is an optional dependency, but because of a programming mistake, a single version goes out where it becomes a non-optional dependency, do I now have to include it (for the users of that version)?
In an even more contrived case, let’s say I integrate it with a vendor closed source program X. The virality is impossible to satisfy, unless I negotiate an AGPL license for X.
> Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication
Intimate data communication is pretty vague. Imagine the AGPL is a database, and I write a custom storage engine. Naively that seems pretty intimate to me.
Either correctly or incorrectly (because it’s never been tested), the perceived virality of AGPL is probably a major reason, regardless of the actual intent.
Also, AWS did offer at least one AGPL service, managed MongoDB. They still offer it, Mongo just changed their license precisely because the AGPL didn't protect them from Amazon in the way they were hoping.
And also, software distribution is different. Typically, you don’t bundle dependencies and instead install them with e.g. a package manager or system library (at least on Linux), so the separation is clearer because you don’t need to distribute the GPLed code to your user (in many cases).
> Amazon offered at least one AGPL service
Are you sure? I only found DocumentDB (which only promises MongoDB compatibility). There’s also a comment by an Amazon employee that suggests Amazon never provided hosted MongoDB when it had AGPL or SSPL [1]. Further down that thread, it also suggests that AGPL at Amazon is possible, but requires extensive review beyond other open source.
No. What URLs the logs are sent to is just a config option - probably not even for the hosted software, probably for the kubernetes pod - not source code. If the logging exporter has to speak a magical protocol to send to Amazon's internal logging, that's Amazon's problem - they can either write a shim service to translate the protocol and then they have to publish nothing, or they'd have to publish their changes that allows talking to the magical internal logging protocol.
> Do I need to include my super secret storage engine X because my modification requires that I use it
If you are hosting the software with your super secret storage engine, then you have to be ready to provide the modified software code to anyone who uses it. If all your users are internal then cool you get to keep it internal - though there's no restrictions on who they can send that code to.
You modified the code to improve it for a use case. The whole point of AGPL is that you if you start distributing that modification then you don't get to keep it a secret and prevent it from being upstreamed. The original project might not even be interested in your changes if it's only for some super specific use case.
In my example, the URLs are not the issue. What I was trying to say, which you actually ended up agreeing with is that they would have to publish the changes that allowed talking to the internal protocol. All I added to that, is that is perhaps not something that they want to share (logging is just an example, big companies have a lot of internal infrastructure for debugging, tracing, monitoring etc).
Re: distributing modifications
You’ve missed the point in my second example. The API portion is covered by my first example. My second example is about possible virality due to the concept of required dependencies.
If I have a special remote storage backend that requires speaking a custom protocol (which is used widely across my existing infrastructure) and I change AGPL code to require it, it is reasonable that I have to publish the source code for talking to the custom protocol (again, covered in the first example).
What is new about this, is that if my version of the binary requires a backend that uses the custom protocol, then just publishing the version of the AGPL software that speaks the API is not enough to be able to run it (because it won’t work without a backend that speaks that protocol). According the provisions for intimate data transfer and executability, it is possible to interpret the license as requiring the backend, which is NOT a part of the AGPL software that you have pulled in as a dependency to be AGPL as well. I assume this is where the concerns about virality beyond the original project arise.
Distributing modifications is reasonable, possibly needing to distribute everything the binary ends up talking to over the network is the concern.
> Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication
If you don't change the source code and you instead write a shim service for it to talk to a special logging protocol, you haven't dynamically linked anything, you haven't changed source code for shared libraries, and you haven't messed with source files.
And even if you tried to argue that "talking over a network" is "dynamically linking" which would be just a completely made-up definition of dynamic linking that would never stand up, it still would not count as something that the original "work is specifically designed to require".
> To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work.
While the AGPL might be untested, copyright isn't, and I don't think any copyright lawyer would say that "calling over the network" is adapting "the work in a fashion requiring copyright permission".
Of course not, or
1. Any web browser would have to be open sourced, and
2. That would be the consequence of an action taken by a third party (eg: me) and not by the parties that created the web app and the web browser.
If that's not how to interpret the license then wouldn't a simple API gateway or proxy circumvent it?
Easy solution: then you just publish your patches for anyone to get. Or just anyone with a login to your SaaS. If you haven't changed anything then they can just get the original source themselves.
Imagine what the web would be like if React users weren't allowed to compete with Meta.
There's two ways that this doesn't seem right to me, though it hinges on the vague term "interacting" and how it's interpreted.
Suppose I use Elasticsearch to power website search on my company's website -- maybe something like a customer support knowledge base of a bunch of FAQs and support articles, and I make some modifications to Elasticsearch to better fit my requirements. My website makes calls to an Elasticsearch service to provide search results.
1. Based on my interpretation of the AGPL, visitors to my site who make searches are not remotely interacting with the Elasticsearch software that I am running; they are not sending requests directly to the Elasticsearch software, and thus they have no rights to its source code under the AGPL. (I'm not suggesting that a proxy server that passes on requests and responses unmodified would be the same situation.)
2. If they do in fact have rights to the source code, it is only to the modified version of Elasticsearch, not "all my source code" (which could include the web server software itself).
> Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. https://www.gnu.org/licenses/agpl-3.0.en.html#section13
> In AGPLv3, what counts as “interacting with [the software] remotely through a computer network?” If the program is expressly designed to accept user requests and send responses over a network, then it meets these criteria. https://www.gnu.org/licenses/gpl-faq.html#AGPLv3InteractingR...
"changes to the product" means changes to the service itself, and "publish all of your source code" means the specific service, not for everything you build. If you patch the ES service you make the patch public, but you don't need to make any service calling ES public. That is pretty static and controlled on your end.
On the other hand "direct competitor" can change over time, so that if elastic buys a competitor to my product or reinterpreters what it means to be a competitor it changes how I can use the software. Say you were early into ML stuff and built a RAG on top of ES, ES will probably offer that soon as a service (if they don't already) so now you are a competitor without any change to your business. Or you want to launch a small project that is ambiguously adjacent to a component (with these non-OSS licenses) that another team within your company uses from a third party. That now becomes a huge legal liability and risk, regardless if you use that exact component to compete with that supplier or are even aware of it.
At least that is the way I've understood the risks of these non-OSS licenses and have gotten similar advice from lawyers at major users of OSS.
I don't think competing with ElasticSearch is mentioned anywhere within the license. If team 1 uses ElasticSearch and team 2 is developing RAG (without using ES), then that's not an issue (but only if using the most recent version of ES, since the license for the code that I fork today has no bearing on future code that ES hasn't written yet).
Hashicorp is just one of the company using it.
- Marketing a project that isn't open source as open source. Debate about what the "definition" is or why it matters all you want; taking a term and using it in a way that contradicts the vast majority of domain experts is bullshit.
- Taking an open source project, which people adopted on the basis that it was open source, which people contributed issues and pull requests to on the basis that it was open source, which people evangelized and promoted because it was open source, blogged about, built on, and so forth because it was open source... and moving it to a license that isn't open source.
To be clear: yes, the unforced error here in many cases is accepting a CLA. That said, I think it's not even unreasonable that people initially accepted CLAs: many of them presumably believed they would only ever be used in good faith, as a sort-of CYA. But CLAs are now very commonplace, so refusing to contribute to any project with a CLA requirement is hard.
If nobody cared about the benefits of open source, then it would be easier for companies to just start with a closed or shared source offering and call it a day; not much backlash for not changing a license. Clearly, marketing something as open source helps... but once you've gotten what you need out of it, it's easy enough to click a button and change it back to being closed.
In my opinion the big advantage of open source is that everyone is on a level playing field. This isn't "fair", it's balanced, and that matters if you are serious about long-term software. If shared-source software is discontinued, that's probably the end of the road for it. For open source software, it only depends on if there are big enough stakeholders to keep funding development; it never has to stop.
There's ideas like BUSL, which might work better... but it's still awkward and experimental. I don't put much stock into any of the other "shared sorta-like-open source" licenses, they're mostly bullshit and sometimes catastrophically horrible, i.e. much worse than AGPL.
For instance, your post calls things "shared source", which, to me, is a lot less clear of a description for the projects you're describing that way. ("Shared" how? Shared ownership? Or what?)
I think "source available" is intuitive and fine (and better than "shared source"), but to me it's still a bit weirder. To me, it sounds like if you send the company an email, they might send you back a zip file with a bunch of source code. But most of these "source available" projects operate just like any other open source project.
But I'm also not unsympathetic to your arguments here at all.
https://en.wikipedia.org/wiki/Shared_Source_Initiative
The software industry called their initiative for what it is. Whether it's "shared source" or "source available", it's a poisoned gift. In the case of Microsoft's shared sources, this was because it was opening up readers of that source to the possibility of patents lawsuits. I remember for instance that Microsoft was making more money from Android, by threatening phone makers with patents, than they did from Windows Mobile.
I have flipped and flopped back and forth on this, but nowadays I think it is worth reconsidering. I think the term "open source" is probably fine and it would be better to actually just double down on it. I'm not sure it could be much better than it is.
What you are saying is largely true: open source is defined to mean much more than what the two-word phrase actually implies intuitively. Fair point, and a common point of contention.
However, that's actually true of lots of domain-specific jargon in general. After all, language doesn't always have a succinct way to intuitively define specific concepts. It evolved naturally over time and surely largely out of necessity to be able to communicate effectively. Every language has blindspots, as well as oddly specific terms you wouldn't expect, like the perennially-cited Japanese term 「青木まりこ現象」(aoki marikogenshō) for the urge to defecate shortly after entering a book store.
When it comes to domain-specific terms, I think we have to accept that the there will sometimes be things where the layperson simply cannot intuitively understand the jargon no matter how its phrased. There's certainly not two words that can accurately explain what it means for something to be "open source" or "free software" according to the champions of said phrases. I mean, take for example, how many words Open Source Initiative has to spend on accurately defining it themselves[1]. Certainly it could be more terse, but no matter how you shake it there's just a lot of detail there.
So what happens is that jargon gets invented where if you know, you know. Sometimes jargon is just bullshit that could be replaced with much more obvious English, but I think often it really is just a lot of domain-specific stuff that can't be described sufficiently with short, simple phrases, so it winds up being bundled into less specific phrases. Does everyone really know what an "operating system" is? I'm not even sure if many computer scientists will agree on a definition for it. Yet, most people agree on which things are and are not operating systems somehow, and it remains an immensely useful term to describe a class of software that virtually everyone, including laypeople, often have a need to describe.
In that regard, I think "open-source software" is about as good as it possibly could be. As far as I could find when researching the topic, it was essentially a completely unused phrase before it was coined, and the people who coined it were very deliberate about giving it a very specific definition and tying it to a very specific movement; and most importantly, they defined rigorously what it was not, which wound up being very important.
I mean, we could call it something else, to be fair, like "free/libre and open-source software" or what have you, but the issue is that open-source is so well-known that it's somewhat understood by people with very little domain knowledge in software. I think the term open source has "stuck". It is true that not everyone really grasps what it means, but I think a lot of people, even if they couldn't define exactly what it means, sort of "get it" anyways. I think that many people who are not software developers have an intuitive understanding for the mutually beneficial nature of open-source software. Don't get me wrong, it's very clear that many people also do not: those people make themselves known in many ways, like being abusive on GitHub issue trackers.
I don't think we can get much more people to understand what open-source software actually is, at least not by force, so I think the better play is to defend the term we have. It's also totally fine, of course, if people want to use "expanded" terms like, again, "free/libre and open-source software", just to make it completely clear what they mean, but I suspect it's just too long and cumbersome to ever catch on the way the term open source itself has, and letting that term get diluted is a loss that will lead to confusion and manipulative behavior.
> For instance, your post calls things "shared source", which, to me, is a lot less clear of a description for the projects you're describing that way. ("Shared" how? Shared ownership? Or what?)
> I think "source available" is intuitive and fine (and better than "shared source"), but to me it's still a bit weirder. To me, it sounds like if you send the company an email, they might send you back a zip file with a bunch of source code. But most of these "source available" projects operate just like any other open source project.
To be honest, I only really use "shared source" because it feels like an analog to "open source". I have no particularly strong attachment to it and would be happy to call it "source available" or anything else. I do have roughly the same feelings though. "Source available" would be a strictly better term overall but I think this all suffers from the same problem that "open source" does: boiling a concept like this down to two words will never be perfect.
> Smolblog shall be entitled to make Your Contributions available under a proprietary license provided Smolblog also makes Your Contributions available to the public under the terms of the GNU Affero General Public License version 3 or later.
It would allow you to maintain a proprietary product with proprietary features that you don't release under the AGPL and use my code within that product.
I like reciprical licenses, if I get code from you under the MIT license, I will give you code back under the MIT license (which you can use however you want to, under that license, just like I can.) On the other hand if you give me your code under the AGPLv3, I give you back code under the AGPLv3 (and you can take it or leave it, so long as if you take it, it is under the terms of the AGPLv3 license).
At least, that is my idealist stance. But in reality, practicality sometimes takes precedence, so I might make a minor bugfix or something. But then I have all the trouble of reading the CLA, making sure I understand it, and agreeing to it, so practicality may just as likely lead me to just file an issue instead and patch my own copy.
> It would allow you to maintain a proprietary product with proprietary features that you don't release under the AGPL and use my code within that product.
As much as I can say "everything in my version is AGPL; this is just for _other_ companies" I don't know that there's a way to _legally_ guarantee it that wouldn't be easily circumventable, at least not without rendering the idea useless in one way or another.
So yeah, thanks for the insight, I really appreciate it!
The Open Source licenses have been vetted and are time tested. That's one big reason for why Open Source is valuable. When you adopt an OSS project, you know exactly what you're getting, and the legal departments of corporations are prepared for it. Some are banning copyleft licenses, of course, for good reasons, but the knowledge is there.
Personally, I wouldn't touch the Elastic license.
> You may not provide the software to third parties as a hosted or managed service, where the service provides users with access to any substantial set of the features or functionality of the software.
It seems risky to use in anything exposed as a customer facing feature
Search may be 10% of your software but what if your software is a managed email provider (or really anything) and you're pretty much exposing Elasticsearch directly through a minimal interface?
It's sad to see elastic turning sides for their benefit, and as a contributor I feel betrayed. While OpenSearch on the flip side is more contributor friendly. I honestly feel all energies should be focused on one product to make it better instead of walking in different paths. Amazon has already taken that path and I don't think they will ever walk back, unlike Elastic.
That's because the license change by Elastic impacted not only Amazon, who could not provide Elasticsearch as a service anymore through its administrative consoles, but also all those vendors who were building APM/monitoring/log-aggregation solutions as-a-service on top of Elasticsearch. In fact, such vendors would typically use Elasticsearch as a back-end behind some custom UI.
So those vendors teamed up with AWS to develop OpenSearch.
Now last time I checked the commit history of the two projects, Elasticsearch had 3x more commits and many of them on cool new stuff, while OpenSearch focus seems to have remained on APM/log aggregation.
As someone who needs an actual "search engine", I am glad of the change, as I was worried OpenSearch may not be a viable open source alternative as it could be lagging behind in this domain.
Now I need to check what happens with the clients: will the client remain Apache License or will they change to AGPL? The latter would be a problem for closed source software.
Not in my experience. AWS is fully behind using opensearch as a search engine. For AI, hard to see how Elastic can compete with AWS...given it's vast resources and deployed products.
Search functions; and
Data ingestion and transformation pipelines,
I do not have the APM/monitoring use case anywhere near in my vicinity, and alarms and monitoring get griggered by / send metrics into CloudWatch.
I await the day that the industry corrects itself and stops calling the AGPL open source/free software. It isn’t. It is very obviously a EULA, despite what the anticapitalist zealots at the FSF wish to claim.
It's really not, because the end users of your service, that being whoever consumes them, does not care about the AGPL and CAN close source their code.
If I call an AGPL service I can do that from a proprietary application. What I can't do is publish an AGPL service, modify that service, and then hide the modifications. So it works just like GPL in that way except instead of, like, including it's publishing an internet-available service.
Companies are super scared of AGPL but that's just because they're scaredy cats (sorry, "risk averse"). But no, you're free to publish an AGPL service and you can even monetize it, if you want. You're also free on the client-side to do whatever and have whatever license you want for your code.
I find what seems to be the prevailing opinion of people here (and in similar places) of passionate opposition to these kinds of licenses to be very mystifying.
It seems to me like they hit a pretty good spot on the continuum of trade-offs here.
I might add one, which is related to your third bullet point, but which I avail myself of far more often:
* If I'm confused by how something seems to work, I can read the implementation.
The four freedoms of software are: the freedom to use the software for any purpose; the freedom to study and change the software; the freedom to share the software; and the freedom to share one’s changes. The AGPL permits all four; the Elastic License does not allow using the software to make a competitor; therefor the Elastic License is not a free software license.
More details: https://www.gnu.org/philosophy/free-sw.en.html
> Who wants to put in years building a business …
Free software is not about the original author of code; it is about the users of that code and what they do with it. Copyleft ensures that those who build upon a software foundation grant the same freedoms to their users which they themselves received. Free software is about the users.
it's perfectly valid to have a completely different view of what freedom is for software
"It's not OSS" is a not a value judgement unless you think that the four freedoms were written by god. But it is treated exactly as religiously.
It’s not a value judgment unless the four freedoms reflect one’s values. They reflect mine; therefore I judge that non-free software does not align with my values.
I don't entirely understand this bit.
> we changed the license, knowing it would result in a fork of Elasticsearch with a different name and a different trajectory. It’s a long story.
I think the name of the fork is now OpenSearch.
What metric are you using to come to the conclusion that OpenSearch is the more valuable brand?
I don't have those metrics or an opinion, im just saying that value is based on utilization by a product's target users, not support activities.
Stars and rate of star growth and stack overflow activity are all passable proxies. They're not great, and I'm open to better metrics, but they're what I can find.
Truly, if anyone can give me any metric that shows OpenSearch ahead I'll shut up. I can't find one, and I've looked.
If I'm right that OpenSearch has a weak brand, Amazon could switch and their internal stats and revenue wouldn't budge.
• Google search volume and quantity of search result pages (blogs, recipes, etc.)
• Mentions in LinkedIn profiles as a skill
• Number of job post listings as a requisite skill
• Social media mentions
It is a measure of "mindshare" or "share of voice." Not one of market share ($$$) or utilization (TBs under management, etc.).
With that said, in the August 2024 listing:
• Elasticsearch is ranked #8 (of 423 systems tracked), with an index score of 129.83
• OpenSearch is ranked #35, with an index score of 16.47
This would make OpenSearch about an eighth as prevalent.
A big question is New + Churning installs. I'd expect a scary portion of the New, and inherently slower rate of Churn are heading to OS instead ES. The curves support that: note that ES isn't substantively growing on that chart. Anecdotally, we see most new installs as leaning to OS in the security industry, which is one of the top money makers for ES/OS.
Opensearch is definitely coming on strong.
That's not true—we're talking about brand value, not financial value of the product. If AWS switched over to offering ElasticSearch again (not that they will) and ditched OpenSearch, I have no reason to believe that their financial numbers would go down a bit.
Brand value is nearly impossible to measure, but to the extent that you can it'd be by measuring perception among those outside the company, not through an accounting of the company's actual revenue.
Think of it this way: the brand LENRUE [0] is worth approximately zero. The company that makes these products could rebrand tomorrow and their revenue stream wouldn't take a hit in the slightest. But the company presumably actually makes some amount of money.
For Elastic vs OpenSearch, the brand value of the two products should be loosely comparable by looking at some measures of public perceptions, and I can't find any measurement that would suggest OpenSearch is in the lead.
[0] https://www.amazon.com/stores/LENRUE/page/24E9713E-AC7E-4269...
Anecdata, but very real.
it happens - why would you think commercial operators who’ve chosen a dual-license model wouldn’t protect their IP? That’s literally their breadwinner.
like, answer the question instead of evading, surely it cannot be that hard.
One of them is not the company talking about their license choice but a FUD article crying about AGPL which we've seen a million tired versions of.
The Rethink one says
> * Require users who choose to modify RethinkDB to fit their needs to release the patches to the software development community.
> * Require users who are unwilling to release the patches to the software development community to purchase a commercial license.
note:
> * who choose to modify RethinkDB
and
> * release the patches
none of these say anything about problems with putting control planes in front of it.
I have worked with cloud hosting a database where the only feature behind the enterprise license is a load balancer with some dead simple authz plugins.
You can write put any LB in front of it and host and sell it with the same capabilities without violating the OSS license. Adding a "control plane" that sits in front of the hosted database does not require you to publish any modifications unless you actually are running a modified version of the open source software. You would never have to publish your own LB.
Edit: As long as OpenSearch doesn’t start breaking the self hosted use case, I don’t see any reason to consider Elasticsearch again. In fact, ES would have to offer up a significant advantage to overcome the bother.
So they were originally Apache-2.0 which was permissive versus now they are AGPL which is copyleft.
The important distinction here is that if Amazon was to use Elastic directly, they'd have to make their contributions available to users and those users could then upstream those contributions back to Elastic. In the old situation with Apache-2.0, Amazon could take contributions from Elastic but then they kept them themself for the most part without up-streaming.
This forces a give-and-take relationship vs a one-way relationship.
Also importantly now that Elastic is AGPL they can integrate anything they want from OpenSearch's Apache-2.0 licensed projects but unless OpenSearch becomes AGPL as well, they can't pull any contributions from Elastic.
There going to be some hard irony here, of making such a fuss about it before, when it was someone else doing it, and then doing it themselves.
We’ll see. Maybe they’re principled enough not to rip off the open search contributions.
If not, you’ve really got to believe there’s no sincerity left at elastic.
I guess time will tell; I’d like to believe they’re better than that.
Now you have two open source projects and one has a copyleft license and the other has a permissive license.
Taking contributions from the permissive to the copyleft project isn't ripping off contributions. It's using open source software and collaborating in the FOSS ecosystem. And Amazon would be free to pull contributions back the other way just as well as long as they agree to the mutual terms of the AGPL (which is by all means a FOSS license).
Contrary to what other people are saying in this thread, I would not say it has better branding than ElasticSearch and that ES has lost its battle. Outside AWS OpenSearch is still not a big contender
Altho i believe it does go back to the license attached to the product.
Besides, I'd be shocked if that analogy is how the law works. Perhaps if you'd bought an individual license then sure, you could resell it with the brand name, just like the car. But wholesale is a completely different situation.
I’d say one is marketing, the other is misrepresentation.
There's not really much actual law at work here, it's all civil matters.
A reasonable consumer or customer might be confused into thinking a service named ElasticSearch(tm) Service is being provided by the company behind ElasticSearch. This confusion is exactly what trademarks rather than copyrights are meant to prevent.
The trademark law doctrine of nominative fair use allows you to describe your product as a hosted version of the ElasticSearch codebase, which provides the substance of the right you were describing, and it's also why you can describe the Toyota car you're selling as a Toyota, both without needing permission from a rights holder.
In the car case, you can also reference the product by name as a Toyota product because it is the same product Toyota sold, just being resold by you. But in the hosted service case, you're not reselling the same service as Elastic does; you're offering your own independent version of the service, backed by their technology. To prevent unwarranted damage to Elastic's reputation from any weaknesses in your service's reliability, customer support, or other factors, trademark law doesn't let you call your service ElasticSearch Service without their permission.
This works similarly for lots of software products and services, even other free and open source software projects. Debian has a trademark policy and exercises oversight of modified / derived / integrated versions shipped by the major public cloud providers to make sure that it's consistent enough with Debian's software freedom values, expected functionality, and quality standards to be called Debian, using trademark rights as the way they have that leverage.
At the same time, the cloud providers do not need trademark permission from Debian to redistribute unmodified official Debian images under the name Debian, or to derive from them without using Debian in the product name. (As with the ElasticSearch example, they can still use the word Debian in a fair and accurate way when describing the nature of any derived product they make without trademark permission.)
Elasticsearch and Redis are private companies that fund most of the development themselves.
When Amazon sell Elasticsearch and Redis, they are in direct competition with its creators.
Obviously such a situation isn't sustainable in the long run, and as such both Elasticsearch and Redis (and to my knowledge also mongodb) have changed their licenses to avoid that cloud providers sell their OSS product without paying a license or otherwise contributing back.
In the case of Elasticsearch and Amazon, Amazon even used the Elasticsearch brand to sell their own version.
As I see it it's a good thing that cloud providers are forced to take part in maintaining the OSS software (forked or not) that they are cashing in on.
Postgres’ license is similar to BSD so even if Microsoft made and was the sole developer and sold it, anyone could distribute or sell it or whatever regardless of who contributes. Similar to Elastic’s old license and why Amazon was legal in what they did.
Amazon (and most all large companies these days) do a ton of OSS dev, but that doesn’t matter. The license of the various software counts.
> you're not reselling the same service as Elastic does; you're offering your own independent version of the service, backed by their technology. To prevent unwarranted damage to Elastic's reputation from any weaknesses in your service's reliability, customer support, or other factors, trademark law doesn't let you call your service ElasticSearch Service without their permission.
I don’t know. The orginal AWS-hosted Elastic Search product was the elastic search code, hosted by AWS. That’s fundamentally the same thing. Maybe the exact wording matters and the service name was “AWS managed elastic search” or whatever. I’m sure the Amazon lawyers knew how to name it.
This feels analogous to “Amazon Linux” where it’s clearly Amazon’s version of Linux (which is also a trademark). Or “hosted postgres” or “Postgres compatible RDS” or any number of other services based on OSS.
Yep, that was the problem, they just called it "Amazon Elasticsearch Service" and it resulted in a lawsuit.
It isn’t, though, when comparing it to the ElasticSearch service offered by Elastic the company. Using the same codebase is very different than offering the same service. At minimum, pricing, billing, support, legalese, etc will all vary between the two. Downsides in Amazon’s service shouldn’t be misattributed to Elastic the company without Elastic’s permission, and trademark law quite rightly seems this more likely if Elastic trademarks are in the name of the product.
As for RDS’s PostgreSQL version, I presume the lawyers very carefully made sure it was called Amazon RDS for PostgreSQL (as indeed it is) and not PostgreSQL for Amazon RDS or PostgreSQL, Amazon RDS Edition or PostgreSQL on Amazon Web Services. Wouldn’t any of these last three names quite reasonably make a lot of people blame the PostgreSQL project rather than Amazon for any failings? Doubly so if the PostgreSQL project offered their own managed service on AWS, as Elastic does.
And as for Amazon Linux, I can think of a bunch of ways they might be operating legally in this regard, but a key one is that they probably have explicit permission from the Linux Foundation or Linux Torvalds to do as they do; in any case, the Linux Foundation and Linus Torvalds are both highly unlikely to sue Amazon for this usage regardless of the legalities, for purely pragmatic reasons that have nothing to do with whether a judge or jury would take the lawsuit seriously, and that don’t apply to companies like Elastic.
No reasonable person is going to think that “Postgres for Amazon RDS” would be materially different to “Amazon RDS for Postgres”. It’s absolute insanity that so much time effort and money is spent on this in so many places to try and skirt around trademarks (which weren’t in place in this case)rather than focusing on the actual details of the product
Was it bullshit when Debian wanted to make sure that Google Compute Engine images shipped as Debian didn't include proprietary software, components that would nonconsensually report telemetry to Google from within the guest operating system beyond what is inherent to the nature of any VM running within Google's cloud, or default configurations that are unjustifiably different in ways that are undocumented and/or would surprise and disrupt the usual expectations of Debian users?
Yes, trademark law is the reason Debian got to have those in-depth discussions and collaborations with Google. Source: I was quite personally involved, both as a Debian developer and as a then-Googler. (And Debian was more pragmatic than you might think about some of the specifics. I was surprised myself.)
Trademark law is not bullshit even though the term "intellectual property" is.
> No reasonable person is going to think that “Postgres for Amazon RDS” would be materially different to “Amazon RDS for Postgres”.
Not in the primary technical nature of the product or the underlying software, no, that's true. But in the overall aspect of whom to blame for the service's faults or credit with the service's strengths - in other words, who is viewed as endorsing the product with the corresponding responsibility for good or bad reputational and commercial consequences - then yes absolutely the naming does change that in a reasonable person's mind.
Imagine you were not a cloud infrastructure expert but rather a stock market investor who sees a lengthy global outage occur in a managed PostgreSQL service running on Amazon Web Services. Further imagine that the PostgreSQL project were commercial enough of an organization to run their own managed PostgreSQL service on AWS in competition to the one run by Amazon, and had PostgreSQL stock traded on a public stock exchange, similar to Elastic's real situation.
In this hypothetical scenario, is it not true that an outage in "the PostgreSQL for Amazon Database" would look bad for the PostgreSQL project and would make the investor likely to sell or short PostgreSQL stock and/or (since the two services compete) buy Amazon stock? And in the same scenario, is it not true that an outage in "the Amazon Database for PostgreSQL" would entirely swap the reputational and financial/commercial impact on both organizations?
(I'm saying "Amazon Database" instead of "RDS" to sidestep the inside-baseball question of whether someone happens to know that Amazon doesn't let external vendors make flavors of the suite of services called RDS. That's an implementation detail that could easily be false at some future time and upon which trademark law cannot usefully rely.)
> (which weren’t in place in this case)
Huh? Elastic definitely had a trademark in place. Not sure why you think they didn't.
> It’s absolute insanity that so much time effort and money is spent on this in so many places to try and skirt around trademarks [...] rather than focusing on the actual details of the product
Why do you assume that? My assumption is that both Elastic and Amazon spend much more on their products than on their attempts to comply with trademark law when naming their products.
> Was it bullshit when Debian wanted to make sure that Google Compute Engine images shipped as Debian didn't include proprietary software, components
Absolutely not. But, calling it Debian on Google Compute vs Google Compute for Debian has absolutely no bearing on whether they’re doing that or not.
Like you, I’ve had similar discussions with lawyers on my side and “the opposition side” and we’ve spent hours arguing over the semantics of these things while ignoring the root problem - in this case google bundling nonfree software with Debian and calling it Debian.
> my assumption is that both elastic and amino spend much more on their products than on their attempts to comply with trademark law when namjng their products
And yet here we are unfortunately discussing how we’ve come full circle, and I’m wondering how many thousands of people hours across google, elastic and all of their users were spent on dealing with a naming dispute.
It does have a bearing on whether Debian has a say in the matter, though. In fact, Google did ship customized images that didn't meet Debian's requirements to be called Debian alongside other ones which did until they were able to achieve good enough outcomes with the latter images - achieving this took a bunch of collaborative work over time.
Those other images were described in ways something like "Google Compute Engine-optimized images for Debian" (I forget the specifics), and indeed Debian completely agreed that they didn't have veto power over the contents of those images, assuming they were in fact for Debian instead of for a totally different operating system.
> Like you, I’ve had similar discussions with lawyers on my side and “the opposition side” and we’ve spent hours arguing over the semantics of these things while ignoring the root problem - in this case google bundling nonfree software with Debian and calling it Debian.
To be clear, Google did not bundle non-free software with Debian and call it Debian - they didn't even do that with their customized GCE-optimized images, though there were ways other than licensing in which those weren't up to the Debian trademark policy standards. Google did the right thing on this issue and only put free software inside all of those image images. But as you might imagine, plenty of people in Debian started out skeptical of that fact until they and Google collaborated enough for the truth to be clear.
> And yet here we are unfortunately discussing how we’ve come full circle, and I’m wondering how many thousands of people hours across google, elastic and all of their users were spent on dealing with a naming dispute.
My hours on this conversation don't count - I haven't worked for Google in almost a decade and am typing this in unpaid personal time.
And, speaking from firsthand memories of collaborating across Debian and Google on this, pretty much none of the discussion was about a naming dispute. Google agreed that Debian had the right to decide whether a modified image could be called "Debian", and Debian agreed that Google had the right use the word "Debian" in the descriptive way I've been highlighting when shipping images not approved to be called Debian.
The nature of the collaboration was much more productive than that: "Okay, Debian wants the images to meet this set of standards to be called Debian and meet Debian users' needs, Google wants the images to meet that other set of standards to be a proper Google offering and meet Google users' needs, some of the relevant standards / workflows / cultural attitudes aren't obviously compatible at first glance, but we acknowledge that we have some shared users who want Debian on GCE to work well for them. How do we achieve this?"
That's not a waste of time at all - and lawyers were not involved in the vast majority of those discussions, because it wasn't a legal dispute.
Similar good collaborations happened between Debian and the Azure and AWS teams, and some collaboration events even happened with Debian plus all three clouds. I must say, 2000s-era me would have been very surprised to see Microsoft hosting an event with Debian developers, giving them \\backslash\printer\paths in order to print something, and offering them Visual Studio subscriptions to enable working on Debian on Azure...
I should probably add a very clear disclaimer here: I am not speaking for Debian, their US fiscal sponsor and legal trademark owner Software in the Public Interest (SPI), Google, Amazon, or Microsoft in any of my comments on this Hacker News post, and while I retain very inactive affiliations with Debian and SPI, I have no current affiliation with any of the major cloud providers.
The confusion between ElasticSearch the software kit and ElasticSearch the hosted service illustrates some of the drawbacks of having the same name for your Open Source product and your business.
Clearly not confused with the original brand, but also advertising a service built off that brand.
But it takes a long time. And it's very costly (especially against a much larger entity like Amazon). Legal battles alone will rarely save you (in time).
[I work for Elastic]
Eg AWS Elastic Compute Cloud (2006) AWS Elastic Block Storage (2008)
I suspect if Elastic tried to take them to court they would have got the trademark thrown out.
Gonna go tell those freeloading kids who take my candy on Halloween that they're exploiting me unless they contribute to next year's candy bowl.
1) Amazon is powerful, thus bad
2) we’re discussing Amazon
3) find something potentially bad
4) use confusing and negative language to throw shade to discredit my target because #1
It’s really frustrating to experience these types of conversation. People explicitly choose to donate their work to the world under an open source license. Complaining they someone uses without contributing is so stupid it defies belief. It’s like complaining because Amazon only pays $5 for a Big Mac when that is the posted price.
Lots can be sad about a lot of this. You can disagree with a lot of this. There have been a million discussion on HN and I don't really feel like repeating it all. But you've spectacularly misunderstood the argument.
It’s perfectly fine to sell your software. There’s trillions of dollars worth of companies that do that.
But I make sure I eat through other methods so I’m able to donate my time.
If Elastic doesn’t want Amazon to use their software, then they shouldn’t release it as OSS. It’s quite simple.
But it’s ridiculous, I think, to claim Amazon is doing anything wrong by abiding by the license.
Elastic shouldn’t feel that Amazon is eating their pie because they chose to put their pie out with a “free pie for everyone” under ASL. If they feel bad, that may be so, but their feelings aren’t as important as what their intellect should set up.
I am just saying your post hugely misrepresented the argument.
That you think the argument is a load of bollocks changes nothing about that.
Not to give out advice, but if your aim isn’t to learn and debate and change minds and be changed, what’s your point? Do you just want to make noise or something?
I would like to properly characterize the argument to understand all sides. Because I want more great software to exist in the world. My belief is that the way to do this is to have people create and share, of their own free will. And I want to learn if there’s a better way.
So correcting your enormous straw-man of misinformation is "noise"? Oh just sod off with your bollocks.
So maybe they should stop releasing future contributions as OSS? Oh, wait…
It’s cool being a programmer because we have such autonomy over our actions and our creations.
But maybe we shouldn’t fund open source development via companies whose entire revenue is selling support for that product? I feel like my favorite OSS projects are ones that are created and maintained by developers working for companies whose business model is based on something entirely separate from the OSS project, but who need the OSS project to support that business. They, and many other companies who have the same need, pay developers to work on the project so they can get what they need from it, but they keep it open source because it isn’t core to their business and being OSS makes it easier and cheaper to maintain.
In this way, there is no conflict of interest between the open source needs and the companies business model.
- ElasticSearch was open-source
- Amazon offered ElasticSearch open-source as a paid service
- ElasticSearch was not happy about this and changed their license
- Amazon forked ElasticSearch (the open-source version) and created OpenSearch based on that, continuing to serve OpenSearch
- (Few years pass)
- Amazon and ElasticSearch are now buddies
https://stackoverflow.com/questions/27793721/what-is-the-dif...
AWS just offered a convenience layer over ElasticSearch
https://www.elastic.co/blog/dear-search-guard-users-includin...
I get it, Amazon is bad, I agree they are too, but not because they’re malicious, Amazon is bad because they’re too large to compete on level ground with anyone other than Google or Microsoft in the cloud.
My peeve is with the companies like elastic that claim they are for open source but they try to prevent the open source from being used as such. It’s a scam to attract developers who care about open source. If I made code I wanted to be open source, I’d understand that means everyone can use it, like a public road or a public park. That includes big corps.
At least Amazon actually supports the Apache licensed OpenSearch product! They don’t even go around acting all superior like those “open source” corps do about it.
Yeah it's not that Amazon stole the code, it's that they were distributing stolen code. It's not as bad but it's still problematic unless Amazon immediately pulled said code when they were notified.
> My peeve is with the companies like elastic that claim they are for open source but they try to prevent the open source from being used as such. It’s a scam to attract developers who care about open source.
I think this was less about preventing open source from being used and more about picking the wrong license for their project. The way I see it they took the long way around because they were afraid of the AGPL.
They dual licensed under the SSPL (which is the AGPL with one change that makes it problematic) and the Elastic License (which they originally also provided code under a previous version of).
Then they are now finally getting around to moving from the SSPL to the AGPL (while technically still offering the SSPL).
Had they gone straight to the AGPL none of this would have been even worth discussing but a lot of people are afraid of that license in the same way people used to be scared of the GPL.
> If I made code I wanted to be open source, I’d understand that means everyone can use it, like a public road or a public park. That includes big corps.
Sure however if you chose an open source license, you probably don't want companies selling access to your software with a few extra closed source bits bolted on without contributing anything back. It's not legally wrong but it's a dick move and against the spirit of FOSS. So even then Amazon hadn't broke any laws but it'd make sense for a FOSS oriented company to pivot to a license they think would force upstream contribution. Elastic just fucked up and chose a bad license (SSPL) because they feared the AGPL. This is just them getting over that fear and picking the license they should have picked from day 1.
> At least Amazon actually supports the Apache licensed OpenSearch product!
They do now. When Elastic was Apache licensed they did not. That was the problem. It was only when they re-licensed Elastic that Amazon open sourced their fork. Had they not, OpenSearch would still be the closed source AWS ElasticSearch.
I disagree with this. Most people use FOSS and do not give anything back, individuals included. The spirit of FOSS is creating things that others will use without compensation. If I release anything open source, it's because I'm donating it as a whole to humanity, including big corps and individuals. I understand that, because I've thought long and hard about what it means to release something with, say, an MIT license. It means you lose having full control of your creation. If I wanted to limit who can use my software, I'll sell it or license it accordingly. Complaining later that your FOSS software was "stolen" or "exploited" or whatever is just sour grapes.
If rando small business or rando dude is just using FOSS software without giving back, whatever people have priorities.
If megacorp is re-boxing it in a closed source manner and making mass profits off of it without dedicating at least some level of engineering hours or money to the project, that's a dick move.
Being unhappy with that situation is totally fine and it's understandable to change the licensing to a more copyleft license to reflect your intents.
That's what elastic did. They essentially went Apache-2.0 -> ... -> AGPLv3 but it took a while for them to figure it out.
And my complaint isn't that people make money off FOSS software or don't give back enough. It's that they make money off FOSS software and don't give back essentially at all while depriving their users of the same rights/licensing terms that the upstream gave them.
The issue that Elastic had is that their entire business model was offering a managed elasticsearch solution. Amazon then created their own offering of the same thing, but of course since it is Amazon it was more tightly coupled with AWS and benefited from being a native AWS solution. There was simply no way for Elastic to compete with that.
Now, there can be a lot of opinions on whether that is a good thing or a bad thing for the open source community, but it should be pretty obvious why elastic didn’t like it. They were a company who had a product they were selling, and then the biggest competitor in the world starts selling THE EXACT same thing with the EXACT same name. They needed to do something to compete.
So they did, and forced Amazon to change the name of their offering to opensearch instead of Elasticsearch. Once they achieved that, they reverted the change.
Amazon first offered Elasticsearch as a managed service in 2015. Elastic began offering managed services in 2018.
https://en.wikipedia.org/wiki/Elasticsearch#Managed_services
That's one interpretation. I've another, which I've seen play out multiple times now across multiple OSS projects: company invents a thing, thinks that because they're the inventor of said thing they'll be able to sell a managed version of it, belatedly realise that inventing a piece of software doesn't magically make you the best in the world at running it at scale.
What Elastic, and most like them, can't compete with is the ability to run highly available/reliable software at the scale of Amazon.
I wanted to pay Elasticsearch to host our search cluster. And I did for a while. But it became clear we were paying for gobs of RAM that we weren't using and we didn't have enough CPU to really cover our search needs. I talked to the head of sales at the time and he said they were working on a plan that would allow us to choose machines that were more CPU heavy but that that was in the pipeline and there was no ETA.
So we switched to AWS and everything worked just fine.
All this is to say, Amazon was not offering the exact same service. They were offering a better service.
I recognize that "open source" is ok with that second way of "using" the code, but I do think it's meaningfully different.
This is very misleading. There is a lot more code than just the kernel for the Linux operating system. If you ran the numbers for a default install of Ubuntu or RHEL that would be far more useful. The Linux kernel requires far deeper skills than a lot of user-and development so it is going to have fewer contributors per year on average.
Source: I am a Linux kernel engineer.
I can't believe they didn't predict that this exact outcome would happen: Amazon forks under a more permissive license and becomes the new standard instead of Elastic because the entire package (managed service from AWS + more permissive license) is a lower risk package to the average business.
I guess you're actually saying that despite their claims otherwise, you don't believe they did predict it?
The truth that I think they are trying to hide is that they are losing adoption to their Amazon-backed competitor and that they will bleed more customers if they don’t go open source again.
I think that when they made the original decision to change the license they thought their product had more pull than it really did. They thought customers would leave Amazon’s managed product for their “superior” product rather than Amazon’s “inferior” fork. They thought people wouldn’t trust Amazon to have the expertise to continue development. In reality what customers wanted was a managed solution from their cloud provider, they didn’t really care if it was Elastic or not.
I totally understand that the situation was a pickle for Elastic in terms of being able to be a sustainable company and perhaps they had to do something. But the thing they did didn’t work, or else they would have stuck with it.
In my experience companies that modify the licensing of their products or get bought by bad stewards like IBM (Hashicorp) or Progress (Chef Software) scare away the engineering managers who make purchase decisions.
Can you share more? What friction did you experience with ELv2?
Also briefly touched on one of the issues here: https://news.ycombinator.com/item?id=41395663
"So why the change? AWS and Amazon Elasticsearch Service. They have been doing things that we think are just NOT OK since 2015 and it has only gotten worse. If we don’t stand up to them now, as a successful company and leader in the market, who will?"
I agree that this is probably not the picture of the huge win they are painting. Still, it must have been frustrating for Elastic to have to explain to potential customers that they weren't reselling an Amazon product.
https://www.computerweekly.com/news/252513588/Amazon-drops-E...
If Elastic the company sold a product called "Elastic Amazon EC2" and it was an API compatible copy of EC2 cloud compute, you can reasonably assume that Amazon would be pissed off about it.
Edit: As was Amazon's choice to offer a service named that once Elasticsearch was established, to be clear.
When Elastic changed the license, all vendors of APM products based on Elasticsearch jumped to OpenSearch.
Maybe, they want to pass the message that the fork, OpenSearch, is enough different to not represent real competition, and in any case, now that it exists, after the investment done, AWS will offer that as an alternative to Elasticsearch and not Elasticsearch itself but managed by AWS as it used to be before the license change.
Now, who wants a managed Elasticsearch service will only buy it from Elastic.
I take some issue with this characterization. Let's look at Grafana in particular. Grafana was not always AGPL, and much of its popularity came before the license change. I've been in multiple organizations who only purchased a license for Grafana to avoid the AGPL terms because it had gained traction already in the organization and switching away would have been more costly, and AGPL software is still outright banned.
That Grafana is still popular does not show that the AGPL doesn't impact usage or popularity, only that Grafana is still popular.
This is just an unforced error to enforce this for a product that is literally an internal analytics tool. You can even host it and sell it as a service under AGPL! you just have to open source your changes/contributions.
The Grafana AGPL-licensed stuff has massive adoption, the few places where corporate lawyers can't get their heads out of their butts can just keep suffering.
Elastic.co is the #1 example we use with our clients when we want to show how 'vague websites make you lose clients.' We show them the website and ask, 'What do you think of this company?' and 'What do you think they are providing as a service?' Not a single client, including tech-savvy ones, has been able to answer.
Elastic.co is probably one of the worst websites that somehow gained popularity despite its crappy pricing model and support. Their documentation assumes you already know everything about their weirdly vague services and have in-depth knowledge of server infrastructures.
To anyone who works for them: If you're reading this, know that your website is so terrible that it became our first example of a crappy company.
The obvious counter example is Linux which has no CLA.
https://github.com/opensearch-project/.github/blob/main/CONT...
Grafana and Elasticsearch with their AGPL, are not deployable for teams that don't even want to provide a competing service because even basic security features (group membership via external OAuth source for example) are only available in Enterprise Edition (TM)
I'm sure there will be people commenting in this thread that they understand exactly what the AGPL requires, and it's not that bad, but their opinion matters much less than the opinion of lawyers.
I've never been able to get lawyers in a business setting comfortable with us using AGPL components, for fear that it will be interpreted at some point to require us to release our application source code.
As a result, we've never been able to use anything licensed AGPL in a corporate setting.
Good news! That license mostly exists: it’s called AGPL.
The one where you have to open source “everything” (not really everything, but close to it) is SSPL [1], but that requirement only applies in certain circumstances.
[1]: https://en.wikipedia.org/wiki/Server_Side_Public_License
> if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version
> To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work.
Calling over the network is definitely not "adapting (...) in a fashion requiring copyright permission". While the AGPL might not be very much tried in court, copyright is.
Let’s take redis, pretending it’s AGPL.
Adding a new command to the source code: definitely a modification.
Adding a new feature to the redis client: also a modification.
Creating a class (RedisPlusClient) that inherits from its client and adds a feature: Maybe a modification? Feels modification-adjacent.
Building a microservice that uses the redis client and exposes the exact Redis API? I don’t know! Maybe? Feels like a technicality that it is “using a network call” so it doesn’t count.
The thing is, I don’t want to think about any of this. I want to build things. We have seen overwhelming evidence that for-profit companies give back to open source, both through contributions and open sourcing their own code.
Are these licenses limiting the actual impact of the projects in exchange for ideological purity? There is consistent feedback from engineers that this is the case.
To the people who do love copyleft, I could imagine them not agreeing with a network being a clear line of modification, especially with how common distributed systems are.
For me and how I read it, I agree with you that it’s not. I also understand lawyers who don’t want to be the test case.
https://opensource.google/documentation/reference/using/agpl...
Elastic yes moreso can be used for user-facing search, but in general the point of these ops technologies is plug it in and let your employees use it.
If you wanna modify and have SSO and re-sell it sure yeah you should expect you're gonna need the enterprise license unless you're gonna host a purely vanilla version and layer your own auth over it, which is also possible.
[0] https://github.com/minio/minio/issues/13308#issuecomment-929...
[1] https://github.com/minio/minio/issues/12829#issuecomment-889...
[2] https://github.com/minio/minio/discussions/13571#discussionc...
Clickhouse has proven to also be a very capable database for logs and there are stacks that use it for log storage.
VictoriaLogs is very promising in this regard!
Users that had to pay the price to migrate to OpenSearch do not have a reason to migrate back to Elasticsearch.
"Amazon is fully invested in their fork." Amazon is a cutthroat business that will change strategy if their investment isn't paying off.
That this scenario isn't addressed at the very top of your "addressing the trolls" doesn't bode well at all.
(I anal, even if I were a lawyer, I am definitely not YOUR lawyer, yada yada)
If I were Elastic, I would require Amazon dot com or anyone else who wants to contribute code to Elastic to sign a CLA. Depending on how the CLA is structured, this could allow Elastic to continue multi licensing?
Elastic have lost the Schelling monopoly on what constitutes the "mainline".
Isn't the whole point that amazon doesn't care about source availability or openness, so long as they can extract profit from people running it?
> [LOVE.]
I don't understand what these are supposed to mean. Is this a new writing style, song references, or just a quirk of the author?
> Forever :elasticheart: Open Source
It's hard to not be sceptical about their motivation.
Too little too late. Cannot trust.
E.g., remember when Heartbleed hit, and the world learned that OpenSSL was maintained by one person getting only $2000/year for it? Fixing Heartbleed was estimated to cost half a billion dollars world-wide.
Agreed, but the point of sharing software is that it's not a zero sum game. The thing you create once is not diminished by me using it many times.
But specifically, those two are examples of companies who started off FOSS, then got exploited by AWS.
Every FS-focused person gives them sh!7, but leaves Amazon off the hook because they exploited under the FS rules, which is an inversion of who actually does harm in the world.
Reference: most software in the Apache Foundation and huge portion of the container ecosystem.
At least there are some security features built in (OIDC/OAuth/JWT/Proxy etc.) which are dead critical to operate any software stack be it internal or otherwise.
As for centralised logging or building a search functionality, OpenSearch was already good enough back in the day at the start of the fork.
I think both ES and OS would continue to flourish in their own ways.
I'm not sure Amazon is willing to go all in to win the war of search services, for that means they need to to handsomely reward the best coders that contribute the most to the OpenSearch project to produce insane amount of high-quality code for better and new features. See the great article Code Hard Or Go Home: https://hypercritical.co/2013/04/12/code-hard-or-go-home. Nonetheless, there's a chance that Amazon may be determined to make OpenSearch catch up with ES, just like Apple has made Apple Maps comparable, even not better than, Google Maps. Therefore, open sourcing ES with AGPL is not a bad choice to retain the talent in the ES community.
And AGPL is kinda restrictive to many large customers too, as the customers do not want to risk being forced to open source their business-critical code. In fact, many companies simply ban the use of licensed software. Therefore, AGPL reflects quite the spirit of OSS while in the meantime will not undermine Elastic's business model.
Regardless of the openness of their code - their observability product is grossly bloated and unimpressive, the security product is sideways, fleet is broken by design, the entire database sector is coming after their analytics use cases at much better perf + much lower costs (and winning), management look incompetent, RAG is a big bet - but unlikely to be the saving grace the stack + company needs. It's truly a product on fire. Elasticsearch was interesting 10 years ago - nowadays not so much. This just seems like a "hope for the best" distraction for scarier things to come for Elastic.
We are there to complain when something becomes proprietary, there's no reason we'd not be there when the opposite happens.
This detail in the post made me chuckle. Oftentimes big vendors give out these kinds of marketing awards strategically.
One big firm I know makes it a point to have its CEO present on-stage awards at its annual user conference to customer that have indicated they might not review.
This could have been the entire article and it would make more sense than whatever this is.
So, whatever that means, is the answer to your question.
Open source communities are essentially anarchist syndicates, collectively working towards common good. Groups like Amazon coming in and taking their work and selling it, profiting to the tune of millions, and contributing nothing back will fundamentally alter the dynamics of the system.
IMO, we need a definition of open source that permits me to limit users to those who aren't actively exploiting me. GPL flavors get close, but not sufficient.
I believe, genuinely, in wellbeing for all, and that we should be working towards the common good of all. Open source is one such effort, but I'm tired of seeing people say, "it's only open source if the code is also permitted to be used against the common good and for the enrichment of a very select few already wealthy people."
What do you think makes it insufficient?
So if I want to ensure my software is used maximally, by the people who need or want it most, I need to ensure there isn't a "gravity distorting player" taking my project and white labeling it to push out all the other people.
The AGPL gets closer, but still doesn't go far enough in defining "modifications to the software" or linking, IMO.
What you probably want instead is "fair source": https://fair.io/
Like their names imply, open source is about source being open. Whereas fair source is about ensuring that code is used fairly.
I would argue that people contributing to open source shouldn't be putting themselves in a position to be "actively exploited". If that's something you're worried about, then open source was probably the wrong choice. You should have sold the code for a profit instead, or established some revenue-limited source sharing (so that only indie devs can use it for free, like e.g. Unreal Engine). Or use a fair source license. Or a proprietary "source available" license.
I'm of the opinion that those people have made a mistake that will work against them, and they should consider revising their definition.
Like, if fair source licenses began to be referred to as "open source", then "open source" will have lost its original meaning. So now when stating that something is "open source", you will have to clarify whether you mean "original open source" or "expanded open source" (or something like that). This distinction will be very important to potential users, since it may or may not restrict their intended use case.
It's no different from if we were to start referring to reptiles as mammals. Now when a biologist wants to refer to only organisms with mammary glands, they will need to use some other term, like "milk-making mammals". It does nothing but cause confusion.
Not sure how else to explain this concept... like, I'm really just talking about semantics and pragmatism here. I don't disagree with you on ideological grounds, if that's what you're assuming.
It's all just people making the best decisions they could. It's clear to me at least that there are existential threats presented by tech megacorps that aren't present awhile ago. Maybe it's time to rethink our definitions.
In a way, their original open source license was suppressing innovation. I know this is not a popular opinion in some circles of pure OSS aficionados but it seems the evidence is to the contrary
The same is happening with Redis after their license change.
You could of course argue that because air is free, it stops innovation in the air trade. But in the end that doesn't end in a sensible argument.
Are there any SMEs that have worked with both OpenSearch (the fork) and ElasticSearch? Are there significant differences?
I know the AWS fork had the big difference back then of having RBAC built into their Kibana portion.
(and before OpenSearch, the AWS-managed ElasticSearch absolutely hurt the ElasticSearch brand because of all of the issues Amazon created - it couldn't even rebalance shards, let alone add new nodes or switch to larger nodes without a blue-green deployment)
Elasticsearch hands down for this area.
There are some Elasticsearch features that were part of X-Pack (their commercial offering) so aren't included in the OpenSearch fork. Some of those features are really nice to have and make life much easier; the enrich ingest processor is something I really miss in OpenSearch.
The biggest differences are in the tooling around Elasticsearch. All the observability stuff, the SIEM features, various integrations, and now the AI fluff. I've worked with clients in different sectors and - aside from the observability stuff (which is really nice) - none have had an appetite for any of that.
The OpenSearch team is doing some really cool stuff and the project has come a long way. I'm sure it'll continue to improve. It has a very loyal customer base and even has its own annual event; OpenSearchCon 2024 is next month!
The exception to this might be vector search, which is a relatively new feature that was implemented on both sides post fork. Lots of users want/need this. And both Elastic and Opensearch provide independent implementations with very similar feature sets. Both build on what Apache Lucene offers on this front. So there isn't a massive difference between the two. But I would give the advantage to Opensearch here since its implementation offers a bit more beyond just the Lucene support.
Kibana had a lot of closed source components before the fork already and Amazon fork of that is a bit more limited. But notably Amazon indeed re-implemented the security layer (before the fork actually), which on the Elastic side is a bit of a dumpster fire of complexity. In general I'm not impressed with the product from a usability point of view. Either on the Elastic or the Opensearch side. But the Elastic version is arguably a bit richer in features.
Some notable recent changes there include trying to introduce a new query language based on SQL and the whole fleet ecosystem (an agent based system) to get logging and other data into Elasticsearch. I don't think either is getting a lot of traction because of the licensing.
“Changing the license was a mistake, and Elastic now backtracks from it”.
We removed a lot of market confusion when we changed our license 3 years ago. And because of our actions, a lot has changed. It’s an entirely different landscape now. We aren’t living in the past. We want to build a better future for our users. It’s because we took action then, that we are in a position to take action now.
I'm guessing these license models are all because they want to just plainly sell their own instances without cloud providers competing with them directly (who typically have unlimited resources to do so!) or keeping any changes or fixes to themselves (I think that was the reasoning with Mongo?).
Offtopic / Meta to the article itself:
What is with the format of this article, like if they meant for the stuff in brackets to be headings, but chose this format instead of making them headings.
I made a career with Solr, but building my product, Wide Angle Analytics, on top of Elastic search, made me realize how much more robust and polished ES actually is.
With restoration of Open Source alignment I am confident we will continue building with ES as we are very happy with it.
I have to wonder how much what is happening with terraform and tofu is related to this.
While I can understand why they went down this path, it burned a bridge and I just don’t know why I would even bother instead of using opensearch.
- https://www.meilisearch.com/
- https://manticoresearch.com/
and maybe
The proof that companies like that publicly under value the success that they owe to being Open Source.
But being able to use the term Open Source, by using AGPL, an OSI approved license, removes any questions, or fud, people might have.
If yes, then it is opensource, otherwise it is not.
I'll never forgive Elastic for locking basic security features behind their paid licence. Over the years probably millions of people had their data compromised due to that (due to people inadvertently leaving instances on the public internet - having auth enabled by default would have helped a lot)
What's the solution here?
The messaging against elastic style licences and even copyleft licenses is too convenient for me to trust as being 100% genuine.
There isn't a 'purer' form of open source which does exactly what you want with respect to big companies using the code.
You can be in favour of licensing that restricts Amazon or Microsoft's right to use your work. But that position is detrimental to, not supportive of, open source, since such a license would not be open source.
Or when they infected their previously open source components like logstash?
That's why most open source maintainers use truly open-source licenses for their software - BSD, MIT, Apache2, and they will be happy if Amazon, Google or Microsoft builds successful product on top of their open-source software, since this gives them the desired recognition.
AWS got Elastic's goodies for free. They just came in and gobbled up all of that value for themselves like vultures. Meanwhile the people putting in the work effectively got robbed.
Small companies should stop doing open source and switch to source available + MAU/ARR restriction clauses.
What on earth? Elastic is a multi-billion dollar company. They are no indie startup, scrappy underdog nor are they victims here.
AWS took the high road during this fiasco despite Elastic's mudslinging and flailing about.
> AWS took the high road during this fiasco despite Elastic's mudslinging and flailing about.
They didn't start as a multi-billion dollar company. In fact, AWS started shipping their Elasticsearch Service in 2015. Public records show that Elastic's annual revenue in 12 months after their IPO in 2018 was ~200m with <1000 employees.
I'd argue that Elastic is a multi-billion dollar company _in spite_ of AWS.
Maybe 20% of the companies I work with have switched.
Smells like opportunity!
I've heard a few anecdotes suggesting some people took it seriously, but while we're sharing those: my company actually adopted ElasticSearch since the license change and never seriously considered OpenSearch.
If you're making the call then that's really one anecdote, not hundreds.
If a "DevOps" company purely provide deployment services, and doesn't offer a managed Redis service, SSPL makes no difference.
It seems that the term "DevOps/services" is a way to refer to companies that offer Redis as managed service, without calling them "cloud providers". This type of service is what a large part of the IT community is againt, and that licenses like AGPL/SSPL try to prevent.
I know the AGPL may be a terrible license and all, but it is allowed by Free Software purists. I hope more companies follow suit.
I also don't think this will inspire a lot of companies or developers to start contributing changes to the Elasticsearch code base again; which is something that ground to a halt earlier. I saw my modest contributions under the Apache license being locked up behind this bullshit license and I learned my lesson: I'm never signing another contributor license again. My trust was violated. Not lifting a finger to help them.
Elastic suffered a self inflicted fork of their developer community three years ago and Opensearch has become the default search solution for a lot of developers and companies. Opensearch replaced Elasticsearch as a neutral ground for open source researchers to rally around. I don't see that changing in any material way because of this license change.
It's interesting that they are doing this though because clearly they are feeling the pressure and basically people using the opensource argument was cutting off their stream of new users. I consult in this space and Opensearch has become the default choice for new users. It isn't even close. Why would you pick Elastic as a first time user? They don't even consider Elasticsearch because it's all closed source and proprietary and Opensearch does the job. I don't think this change is enough to change that.
IMHO their next logical step is embracing/acknowledging Opensearch and moving their efforts to join Opensearch and supporting that. That's a huge community of users, developers and companies that's just sitting there without delivering any revenue to Elastic. It's stupid; they are competing with their own product and leaving a lot of money on the table. Elastic has all the core skills to support that community but they are just sitting on their hands now pretending it doesn't exist. They must be starting to feel the pressure to just toss in the towel and grab a chunk of that market. This our way or the highway position sure has resulted in a lot of people choosing the highway.
The point is, having Opensearch as an alternative is finding answers to my question to resume Elasticsearch development not easier than before. All the positive aspects are still counting for Opensearch. It is true that AGPL is not enough.
While I do not think there will ever be an effort by the Elastic company to join Opensearch or embrace the Opensearch community, it seems to me the license switch to AGPL was driven by the analysis that Elastic's product offering can not be copied any more by hyperscalers. It was a mere question of the exclusiveness of the cloud service offering. That collided with the Apache license once. But now, it is clear that Amazon will never switch back to the Elastic stack as their primary cloud service search product.