Elasticsearch is dead, long live Open Distro for Elasticsearch | Hacker News Reader

Elasticsearch is dead, long live Open Distro for Elasticsearch

logicalclocks.com

103 pointsnathaliaariza59 comments

Rephrased: Amazon played hardball with open source Elasticsearch, limiting elastic's ability to make money and forcing a license change. Now you should use Amazon's fork instead!

This is not a good outcome for people who want an independent OSS ecosystem.

My PoV: a bunch of startups chose to do open source because it’s good PR and marketing. Then it proved to be bad for their business model so they moved to something much closer to shared source.

Personally, I don’t think there’s anything immoral or shameful about selling proprietary software or shared source or whatever. However, SSPL and TSL and friends are just not open source licenses. They’re more restrictive than AGPL. They are designed to protect a business model, not your rights as an end user.

So it irks me that the only perspective I see reliably is that Amazon is screwing everyone and not how startups are willing to take advantage of the good will and marketing bonus and then switch it up when they realize it can’t be reconciled with their business model. Continuing to call it “open source” is just insult to injury, imo.

Basically shareware is back, they just keep calling FOSS.

> not your rights as an end user.

I'd love to have a license that grants FOSS rights to people but not to corporations. There are end users and then there are corporations monetizing FOSS and giving nothing back. On top of that the monetization is via SaaS which is a more closed computing model than commercial software. With SaaS you don't even have the data and everything you do is subject to total surveillance.

creative commons has many such licenses but then it's not Open Source

The point about not giving back is not valid in the case of AWS which has upstreamed many of its changes to elasticsearch owned repos under the onerous elasticsearch CLA. This is also true of many FOSS projects which AWS, Microsoft, Google et al contribute to under the most liberal license conditions(such as MIT licensing which you'll find in 95% of Microsofts contributions for example). Furthermore, I do not believe FOSS should concern itself with who is using the codebase, as fully open source software is about opening the code indiscriminately, whether individual dev or large corporation. As soon as you begin to discriminate between small and large corporation you are no longer truly free (and fwiw elasticsearch itself is a large corporation that has arguably exploited the good will of individual dev PRs with relicensing under a non-FOSS license).

Shouting 'FOSS' is also a great way to acquire talent.

I think you're painting it as more cynical and malicious than it was. There was probably too much naivety, but "open source your software and charge for support and operations" was promoted pretty heavily for a while not just by these companies, but by people who wanted to enjoy those projects being open source.

And then it got torpedoed...

"Bad for their business model" because Amazon has tremendous power. It wasn't just a series of bad choices by the startups that got them here, it was concentration of an emerging market over 10-15 years.

Amazon dominates conversations because they extract the most value from open source projects. They make more money selling PostgreSQL, MySQL, Elasticsearch, _Docker_, Linux, etc than the creators of those projects. It's not the only perspective, but it dominates for a reason.

Startups OSS their work because it helps them get users and figure out what to build. Amazon has the benefit of watching which projects become most viable (frequently because the startup threatens their revenue somehow) and packaging them up as their own services.

I want to live in a world where smaller startups _can_ OSS their work and also monetize it. And I'd like them to be able to do it without raising $100mm in VC.

>"Bad for their business model" because Amazon has tremendous power

this goes to the question of history, are things structurally so determined that if you remove someone or something from history another would rise to take their place. If Amazon had not existed to take advantage of open source like this would someone arise to do it? I have to think yes, it would be bound to happen eventually, it's a weakness in the model and would be exploited by someone, and when it was exploited people would respond by doing sort of like what is happening now.

If Amazon was not here, someone would be making the history of open source exploitation rhyme.

> They make more money selling PostgreSQL, MySQL, Elasticsearch, _Docker_, Linux, etc than the creators of those projects. It's not the only perspective, but it dominates for a reason.

They make more money selling EC2, with this software installed on top of it. If Amazon did not sell RDS, their customers would buy EC2 instances and install it on there.

Which of these realities would you prefer:

1. Mongo, ES, PG, etc have a cloud offering, which runs on metal they maintain. Naturally, the reliability and performance is worse. Amazon loses some money. Customers pay more for a worse product.

2. "Open source has failed, there's no way to monetize it", so we get DynamoDB and Aurora and S3. Amazon wins, Customers (mostly) win (unless Amazon takes the gun they've now loaded and decides to point it at them)

3. "Its open source, but you still have to buy it" lol ok, all the negatives of closed source with few of the positives of open source. No one wants the source code so they can read it and hack on it, they want it only as a signal for their ability to move hosting providers should they need to, or self-host. Source code usually lasts longer than the people who wrote it.

4. "We'll sell support packages" so now your revenue stream isn't aligned with customer success, because you're financially incentivized to build a product which needs support. Nope.

5. AWS does not have RDS, but does have EC2. Mongo, ES, PG, etc have their own "cloud" offerings, which are really just re-vendored EC2. They tack on a 30% upcharge so they can survive and develop the product. Well, that's what both ES and Mongo do, today. AWS loves this. They make the same amount of money from selling cheaper metal without all the expensive upkeep of RDS. Its the customers who hurt, not Amazon. You miss out on integration points in the cloud platform. You miss out on centralized IAM. You pay more for a worse product.

People keep acting like AWS is a big bully who got big for no reason, it doesn't make sense, the cloud is a monopoly and AWS is strong-arming other companies to get bigger. Maybe to some small degree, but by far the reason why AWS is big is because They Build What Customers Want. End of story. There's no philosophical debate between AWS Product Development and their Customers concerning whether they should build an open source ES. There's just "Jeeze, Elastic has been really screwing us over lately, is there anything you can d--" "Say no more, we've got you, by the way your two hundred thousand dollar bill is due." "Wow, thanks Amazon, no problem we'll pay that right away."

The real thing that's killing these database companies is that databases are a dime-a-dozen. You pick one, who cares which one it is. There's some minutia here and there, do you want SQL, NoSQL, time series, ok you get past that and the rest falls into place. I've sat in the decision making room on this for two or three companies. No one analyzes what the database can do. The discussions surround "where are we hosting it". What options does Azure have, AWS, oh that company has their own hosting platform, etc etc. HackerNews makes a big fuss about how MongoDB has no transactions; I've literally NEVER heard this brought up once as a negative against the product (though, sure, in some very specific industries or use-cases it matters). Version 4.whatever adds a new query operator; who gives a fuck, 99% of database usage is "write thing, read thing back, query for things in a way that SQL could do in 1992, update thing, delete thing." Databases are boring, and the success of the companies who tried to build billion dollar businesses on top of one (Oracle as well) reflects how boring the software actually is.

I feel like [3] is the heart of the matter. The value is open-source to companies is that it commodifies support. Also software at an enterprise scale costs money. Not being held hostage allows you to switch support providers or go internal or fly blind.

What these new quasi-open source (ie source available) companies want to do leverage their authorship of the software to be sole support service provider (of meaningful size)... which is exactly the sole value the enterprises extract from opensource: the lack of vendor leverage!

The market does not tolerate a contradiction.

Exactly. Companies like Mongo, ES, etc leveraged open source and venture capital to get where they are today, and now they're finding that open source isn't the business that venture capital wants it to be.

I really don't know why they get so much defense in the community; I view their actions as a straight betrayal. The one thing you can say about AWS is, they're the enemy you know. They don't have a history of turning their back on OSS, but they also don't have a history of abusing OSS to gain traction for their product (though, the MySQL/Aurora relationship could reasonably fall under this. I think its different; not a betrayal because they haven't turned their backs on MySQL, but are just trying to make it better for their customers in ways MySQL alone can't do). With AWS, you know what you get; they care about the tech and the money; no politics (well, almost), no weird philosophies guiding decision making, just solving problems and getting paid to solve those problems. That's infrastructure, baby.

Its a lot hazier with, say, MongoDB. You wouldn't believe the pressure they've put on customers on 3.6; we're talking weekly emails, even a few cold calls, from sales reps. "Hey, I know its the holidays, but have you taken a look at 4.0 yet? Let me send you a link to check it out, lots of cool stuff." Yeah; the real reason they want you to upgrade is because 4.0 has a bunch of new features that competitors like AWS can't replicate due to its new licensing. I'd love to hear from someone running MySQL on RDS, and if they've had a similar experience with AWS sales concerning Aurora; I suspect they haven't. Frankly, AWS never talks to us and we never talk to them, unless there's a problem or we want a discount, and they're always accommodating for both. The perfect relationship.

> Startups OSS their work because it helps them get users and figure out what to build.

Therein lies the problem: Startups esp the ones that need to grow really fast and at all costs choose to permissively give away their proprietary advantage but still want to "capture most value" from the market they help create are making it unnecessarily hard for themselves by going down the F/OSS route just to "get users and figure out what to build". It is a losing strategy (you'd agree?) because... if the startup's F/OSS product is any popular and lots of users do use it, it is only inviting the likes of Amazon (and anyone with the chops to build a cloud offering, really) to package it as SaaS / PaaS around it.

I'd like to see startups experiment SSPL / CCPL / Commons Clause licensing from day zero (like materialize.io) to see if they can still attract users or build mind share. I'd hate for them to instead release it under F/OSS licenses (and CLA all contributions) to only conveniently turn hostile with super-strong copylefts like SSPL or non-Libre licenses like Commons Clause and BSL.

This is exactly why I convinced my company to buy Airflow from Astronomer and Presto from Starburst and Spark from Databricks instead of getting all that from AWS.

I would rather give the money to the people who make the stuff than to people who (all but) steal the stuff and sell it to others.

And it usually comes out cheaper, too!

I tend to think that it, more or less, is. Elasticsearch (as an open source product) is not independent. Its reliant on Elastic co, who have a demonstrated history of reducing its FOSS-stance in favor of making more money. Of course, they had to do this; Amazon forced them into a corner, similar to MongoDB. But the fact that there is a critical company behind these two databases, who can be strong-armed, is a threat to the ecosystem these two databases have developed.

Compare to Postgres. Postgres will never die. Aurora is gaining serious traction; do you think the Postgres developers are concerned? Why would they be? Their livelihood, and their ability to develop Postgres, does not depend on it being the #1 database in the world.

Naturally, ODfES has a tether to Amazon. Amazon could decide to lock more features behind AWS ESS. Its not a perfect situation, but it is better than the one ES has been in for the past few years.

I think having a company behind it would generally be a positive. People are paying for this product to continue, versus something with just a couple devs doing it as a hobby.

Postgres had a huge head start here, and was built in a different world WITHOUT an Amazon equivalent.

Are there a lot of major contributions to (not just on top of) ODfES outside of Amazon? If not, I'd be fairly worried about using it in a non-AWS-provided way in the long term, now.

You say that: "Elasticsearch (as an open source product) is not independent. Its reliant on Elastic co, who have a demonstrated history of reducing its FOSS-stance in favor of making more money. Of course, they had to do this; Amazon forced them into a corner, similar to MongoDB."

But there is one more factor you failed to acknowledge that is driving them to make more money: Elastic.co took VC funding.

Once you accept VC funding, it narrows the definition of success down to what your investors think success should look, like rather than what the founders desire.

This is why lots of more serious projects have been going to semi-OSS "source available" licenses like the BSL or more restrictive licenses like the AGPL. If you're a company doing open source or a serious project and use a liberal license, your work will get co-opted and monetized by cloud vendors and they will give you nothing. In some cases they won't even give you credit.

> This is not a good outcome for people who want an independent OSS ecosystem.

I used to manage a licensed ELK cluster. The licensing was expensive (order of six figures) and was really only necessary for LDAP integration. Features like timelion were cool, but definitely not worth the price tag.

I migrated to an AWS-managed solution to for two reasons: 1) to alleviate the burden of managing a cluster, 2) get away from the onerous costs associated with licensing. Even thought it was more expensive to run a managed cluster, since, at the time, AWS limited the disk space per node. The cost savings was significant enough to warrant migration.

I realize that Elastic.co offers their own managed solution. But the version on AWS was an easier sell to management due to known costs and not having to deal with sales people.

I think a lot of OSS is having difficulty transitioning to a cloud-first model. The people making choices on tech stacks want to spend less time dealing with infrastructure and more time developing stuff. And this model of charging high licensing fees for "enterprise" features feels antiquated, and provides a large attack service for companies like AWS. What enterprise engineers really want are AMIs, or better yet, fully managed cloud interfaces, to bring up and manage OSS deployments. Something that lets us focus on the products we sell; not the ones we buy.

I believe that we need to rethink how OSS is delivered and how to make money from it. "Enterprise" support looks a lot different not than it did when Redhat was founded, yet the OSS business model remains eerily similar.

It makes _total_ sense to migrate to AWS's Elasticsearch product. This is, I think, the problem with incredibly powerful companies. Acting rationally as consumers makes them more powerful.

I mean, I still buy stuff from Amazon, use Gmail, etc.

We definitely need to rethink how OSS is delivered, but part of rethinking that is understanding how dominant AWS over OSS revenue.

> not having to deal with sales people.

This aspect does not get nearly the attention it deserves.

Unless your product is an exceptionally well focused developer tool, it will take months, and sometime multiple quarters, to see whether it adds any real value. Exceptions exist but are rare. In practice the generous[TM] 30-day trial period is almost always worthless, and often a gateway to sustained net negative.

Add to that the constant barrage of dark patterny attempts to infest your calendar and the upsell pressure tactics. Every third-party integration is an additional liability for maintenance, and far too frequently an increase in attack surface. Time is your most precious asset and the sales people are trying to rob you of it.

To every sales person on the planet: if you want to sell me a service or product, then sell me NOTHING BUT the service or product. Make useful documentation accessible, and make additional features discoverable, with unambiguous pricing clearly in place where necessary. Then get out of my way.

With OSS product I can install and trial it as long as I need, without needing to give off my contact details or be added to your invasive CRM. If I need additional features that are only available under your more restrictive license offering, I will buy them when I have the need. Your sales quota or bonus thresholds do not factor in, and if you try make them my problem, I will look at migration options.

> This is not a good outcome for people who want an independent OSS ecosystem.

I understand why this is not ideal for Elastic-the-company, but is it a bad outcome in the longer term?

As someone who is happy open source exists but uses mostly proprietary software, this seems like a natural and expected response to an open source project adopting a more restrictive license.

I think we're all better off if small and midsized OSS companies thrive. Amazon is not inventing and open sourcing DBs. For the last 15 years, we've relied on independent companies to advance the state of the art for _all of us_. Amazon has no reason to build a compelling database and release it to the community. When the giant tech companies do open source they're work, it's usually an attempt to improve their existing market at the expensive of another company: Kubernetes, OpenDistro, Firecracker, and most of the CNCF projects follow this pattern.

A fork is a natural response to an OSS project license change. But when the company that _forces_ the license change also creates the fork, it's gross.

Great points and a helpful perspective, thank you.

Maybe somebody who knows the history can answer this question: how was it that Elastic didn't just own the cloud service market before the AWS service showed up? They started in 2015 it appears. That should have been a good lead. [1]

[1] https://www.elastic.co/blog/announcing-support-for-elastic-c...

Edit: added source

By fork they mean: a binary build straight from the unmodified elastic source code using the elastic build file that creates the OSS build bundled with a handful of plugins and components to add alternatives some of the things elastic bundles. They don't even build it themselves and it gets downloaded in binary form straight from Elastic's servers.

If you don't believe me, this is the line in the code where they 'fork' in their build script: https://github.com/opendistro-for-elasticsearch/opendistro-b...

It will be interesting to see if opendistro actually starts contributing to Lucene or adding value to the elastic code base or whether this 'fork' is effectively frozen in time and doomed to go nowhere due to a lack of actual development on the core product.

I seriously doubt that they are going to put any effort whatsoever in that. Because in two years of pretending they have a fork, there has been no serious code contributions from their side at all that I'm aware off. Correct me if I'm wrong.

If you want to back an OSS version of related technology, maybe use something like Apache Solr, which continues to provide many of the same features and is part of the same ecosystem of users and developers that work on Lucene, which also powers Elasticsearch and which of course continues to contribute a lot to Lucene; unlike Amazon.

Disclosure: I work for Amazon, but not directly on search related technology.

Some information about Amazon's investments in upstream Apache Lucene can be found here: https://aws.amazon.com/blogs/opensource/amazon-giving-back-a...

Covered in the 2019 blog post are:

  * Concurrent updates and deletes
  * Indexing custom term frequencies
  * Lazy loading of Finite State Transducers

PSA: This site is basically an ad. And not one about Elasticsearch at that.

I don't understand what all the noise is about. The new license does not make ES closed source, as far as I can tell. They re-license under the MongoDB server side license, which reads a lot like a rephrased AGPL to me.

None of you had any reasonable expectation of getting a free Elasticsearch to begin with, and decades worth of free maintenance, optimization and expansion.

If their product is this integral to your business, then may I suggest going ahead and buying a license for it?

> None of you had any reasonable expectation of getting a free Elasticsearch to begin with

That is sure what "Apache 2.0: now and forever", as they previously promised, means to me.

Me too long ago.

Nowadays I'd instead interpret that as "security updates for the current version forever",

... after so many smaller companies having gone SSPL or something like that

For what it’s worth, SSPL is NOT like AGPL. IANAL but...

AGPL is community oriented. With AGPL you have to contribute your modifications back even if you are only offering the application over the network and not in binary form. It extends the copyleft principle. It is still debated whether it even counts as open source.

SSPL is fascinating. Let me quote Section 13:

> If you make the functionality of the Program or a modified version available to third parties as a service, you must make the Service Source Code available via network download to everyone at no charge, under the terms of this License. Making the functionality of the Program or modified version available to third parties as a service includes, without limitation, enabling third parties to interact with the functionality of the Program or modified version remotely through a computer network, offering a service the value of which entirely or primarily derives from the value of the Program or modified version, or offering a service that accomplishes for users the primary purpose of the Program or modified version.

> “Service Source Code” means the Corresponding Source for the Program or the modified version, and the Corresponding Source for all programs that you use to make the Program or modified version available as a service, including, without limitation, management software, user interfaces, application program interfaces, automation software, monitoring software, backup software, storage software and hosting software, all such that a user could run an instance of the service using the Service Source Code you make available.

Emphasis mine. Basically “release the source for your entire stack.” That’s not the same thing as AGPL. Copyleft is concerned with linkage. This license seems to impose terms on entirely unrelated pieces of software. IANAL but I don’t even think this has been tested in court before.

“release the source code for your entire stack”.

It’s not “your entire stack” but rather your entire stack:

> If you make the functionality of the Program or a modified version available to third parties as a service

...

> Making the functionality of the Program or modified version available to third parties as a service includes, without limitation, enabling third parties to interact with the functionality of the Program or modified version remotely through a computer network, offering a service the value of which entirely or primarily derives from the value of the Program or modified version, or offering a service that accomplishes for users the primary purpose of the Program or modified version.

Where:

> “The Program” refers to any copyrightable work licensed under this License.

At the end of the day this is Copy-Left debate all over again

Personally I like it, I see "Open Source" being killed off with Apache, BSD. MIT and other non-copy left licensing, it allows companies like Google, Apple, Amazon, Microsoft and other to leach off of "Open Source" focusing exclusively on dev tools and libraries used to build their products.

This has lead to ALOT of developer growth, and ALOT of Open Source access for Developers but no much growth in END USER software products that are open sourced.

Linux one of the most popular Operating Systems in the world, and is the final product not a development library is as big as it is because of Copy Left, one can make a big case that the BSD's never got to be as big as Linux directly because of their permissive license.

Indeed, the proof being how much they get back from Apple, Sony.

In fact, had Linux not been around, most likely all major UNIXes would be still be around, using pieces of BSD as always.

After bending the knee at the altar of Open-Source, the blog post recommends Open Distro for Elasticsearch's Security plugin.

ODfE's Security plugin is a fork of floragunn GmbH's mixed-license Search Guard product, including the bits that floragunn charges for. This seems to have been done with floragunn's blessing - presumably Amazon paid floragunn for this.

But Elasticsearch B.V (the company) accuses floragunn of stealing Elastic's non-free X-Pack security software to create Search Guard, putting both Search Guard and ODfE's Security plugin on shaky legal ground.

I remember reading about this lawsuit on Elasticsearch's (admittedly slanted) blog, not sure how far the lawsuit has gone from there: https://www.elastic.co/blog/dear-search-guard-users-includin...

Amazon of course says that Elasticsearch's claims are baseless: https://aws.amazon.com/blogs/opensource/launching-open-distr...

In either case, I'm not sure that ODfE will be trouble-free in terms of licensing issues, depending on how these lawsuits go.

Open Distro is not a fork (not yet at least) but simply a repackaging of ES plus some new plugins.

Open Distro will likely not continue to exist after this license change unless Amazon commits to turning it into a complete fork starting from the 7.10 version of ES.

Not too long ago, I tried to install the ELK stack to enable me to ingest some log files and easily view them on a web page. After many hours of digging through documentation, I gave up. The process was a train wreck of obscurity, complexity and heavy weight processes, just to do a relatively simple task.

I've been doing open source for 20+ years now and one thing that we still haven't seemed to get right is: a simple install process and good documentation.

ELK is dead to me, but not because of a license.

I've been a mostly happy user of Elasticsearch for 5+ years, but agree it is a bit heavy for just logs (which is ironic, since that's the use case that really launched it beyond application search). I recommend Loki for logs, especially if you're already using Grafana: https://grafana.com/oss/loki/

Years ago, ELK was the thing for log files. logstash. https://logz.io/blog/server-log-analysis/

loki is indeed what I ended up with and it was a mostly easier install process (despite the missing easily installed debian packages).

The next problem being that Graphana is actually quite terrible at rendering log files. It seems the log file usecase just isn't that popular.

https://lucene.apache.org/solr/

Last time I was exploring this space, ElasticSearch was hands down better than solr from a "speed to market" and "developer experience" standpoint. Those things really matter. It's one of the reasons MongoDB got way more popular than it should have based on all the compromised built into the platform by default.

It's still 100x easier to setup a relocated MongoDB cluster with auto failover than it is with Postgres or MySQL. It's an underrated power.

I'm also a fan of projects like fuse.js which are pretty capable little search engines. It's nowhere the scalability of elastic, but if you have even a few thousand documents, you can jam them into an index the size of a big jpeg and have pretty capable search with no infrastructure.

The bad news is that anyone who has hitched their wagon to ES almost certainly has a bazillion lines of JSON query language for interacting with the thing (unless they stick strictly to the [query_string](https://www.elastic.co/guide/en/elasticsearch/reference/7.7/...) search language, which I believe is lucene-compatible

That's not even including the meta APIs like /_cat/indices and /_mappings et al that further lock one into ES-isms

This is why the Apache Software Foundation exists. The threat IMO isn’t SSPL but what happens the next license change. There’s nothing stopping future changes to make Elasticsearch proprietary. It’s not out of the question that Elastic could be acquired by, say, an Oracle. Or there could be new management. Suddenly they change licenses of take a ridiculously maximal view of SSPL or Elastic license and sue everyone.

With ASF, at least the trademark stays with the Foundation. Along with an effort to build a health community spanning more than one company.

Elasticsearch has many benefits. But you have to take this risk into account if you go down the Elastic path now. I wonder if open distro though could become an ASF project, alleviating fears of license change...

>Elasticsearch is dead

No, it is not.

Is anyone aware of an open source alternative for Kibana and Logstash (based on lucene)?

Loki and grafana? (not really an alternative in the sense of "full text search" over logs, but an alternative in the sense "an interface for aggregate search over logs").

There's also Cloki, which offers the same Loki APIs but stores data in ClickHouse. It looks interesting.

https://github.com/qxip/cloki-go

Disclaimer: I work on ClickHouse at Altinity.

It does look interesting, but maybe a little too exciting: "Super experimental". One advantage of loki, is that it is part of hosted grafana, so there's some commitment to it as a product.

Grafana started out as a fork of Kibana 3 if I remember correctly, which was around the time Elastic acquired Kibana. Since then, it has become a nice product in its own right. But the shared history is interesting.

https://grafana.com/blog/2019/09/03/the-mostly-complete-hist...

Elasticsearch, Kibana, and Logstash all have OSS distributions, and are included as part of Open Distro.

Fluentd (https://www.fluentd.org) is a great alternative to Logstash that I've had a lot of success with. It is easy to pair with Elasticsearch and Kibana and myriad other inputs and outputs.

I've personally found Fluentd to be flaky at scale or under heavy load, but it does talk to damn near everything and is pretty simple to stand up