Amazon Elasticsearch Service Is Now Amazon OpenSearch Service
aws.amazon.com
aws.amazon.com
https://s2.q4cdn.com/265747582/files/doc_downloads/2021/Elas...
Revenue for FY2018 and 2017 was $160M and $88M respectively.
Is that what Amazon taking all their revenue looks like?
However, when competing with a literally A in FAANG, one must be very careful about writing them off.
To compare, I knew a company that 10 years ago made really great business intelligence integrations into Excel. They'd operated for 15+ years and were a major local employer. They thought Palantir was going to be their competitor, but then Microsoft came out with Power BI.
At first, revenue grew because Microsoft's marketing shined a great spotlight on the field (plus MS had only entered because the field was growing in the first place). Then as the situation matured, MS's capacity to offer integrations to the rest of the stack became a key sales point.
Like AWS, the point of investing in their stack isn't one particular product but the fact that all the billing, support, and integrations happen via one company.
You can even accept it if any particular product is deficient (e.g., some say that about Teams or Skype), but in the case of an open-source product like Elastic--the value of the software itself is 100% the same.
AWS was going to eventually kill Elastic. Not just because of all I wrote above, but also because Elastic's got a significant part of its revenue by selling Elastic as a distribution on top of AWS. Anyone would want to cut out the middleman and simply use AWS's version even if it lagged behind the current Elastic.
That is so twisted.
Elastic did all of the hard work. Amazon came in and took it.
Open source is fair when there aren't giants plucking it out of the hands of workers and co-opting it into their massive platform options with god-tier leverage. You can't compete in that world.
We need licenses even stronger than the AGPL to require that platforms running the code are open all the way down to the billing stack.
I think it is hazardous to count the number of committers and PMC members that are part of a project that is built at the Apache Software Foundation. It is helpful to keep track of these things to make sure that the project keeps its independence, and is not unduly influenced by the interest of any one company or organization. Committers and PMC members earn their position based on their own efforts as individuals. They don't represent their employers.
The current Apache Lucene PMC chair just happens to work for Amazon today. A number of folks that used to work for Elastic now work at Amazon. But over-focusing on that, and keeping careful track of what company contributes how much to ASF projects, is bad for community health in my personal opinion.
See also Doug Cutting at https://twitter.com/cutting/status/1359160158723711002
Companies should also be respectful of the communities that produce and maintain software, including understanding the philosophy and governance that communities have adopted. For the Apache Software Foundation, this is being mindful to not unduly influence the project, such as to try to control it through employing a majority of the people who work on the project, or otherwise weaken the independence and autonomy for the community that builds and maintains the software. [2]
I've seen situations in the past where companies employ an open-source marketing strategy that ends up encouraging unhelpful activity in open source communities. This can also happen when companies measure the performance of a software developer based on metrics such as "number of patches accepted in an open source project." This kind of thing is generally unwelcome in open source software communities. See this thread on HN [3] on the Linux kernel development list, TL;DR - "Please don't waste maintainers' time on your KPI grabbing patches."
[1] https://blogs.apache.org/foundation/entry/the-apache-way-to-...
I’m sure this will change how every other company is the OSS world will operate from now on. CockroachDB for example is already preventing a monopoly like Amazon from taking over.
1. All of which is useless without Lucene. In fact, it wouldn't have been possible without Lucene. Thus, by your own twisted argument, Elastic "stole" Lucene's work.
2. Not all of the code in ElasticSearch is written by Elastic. Co. It's not their sole codebase. They have no monopoly on it. Maybe if they never took in contributions from others, but they did.
> And Lucene is still OSS.
OpenSearch is still OSS. The ES base that OS forked from is also OSS.
> ... a monopoly like Amazon from taking over.
Now I know you're being dishonest. Amazon never had a monopoly on ES, it still has no monopoly on OS. It's Elastic (like CockroachDB, that you mention) that are trying to enforce an artificial monopoly. Of themselves.
If you want to find out whether ES is really the "little guy" or the "monopolist", try providing a hosted service of their latest release and see how they react. It doesn't matter if you've contributed code to ES before, it doesn't matter if you're the most prolific contributor to Lucene.
It doesn't matter if you can build a better version of ES or provide better service. Elastic Co. (and CockroachDB Inc.) will simply not allow you to compete.
----
For once, try pointing the finger fairly, at everyone. Try judging everyone with the same set of laws/morals. Hold your heroes to the same standards, at least, as your villians.
Also your arguments doesn't make any sense. Lucene is still OSS and OpenSearch is not Lucene but built of Elastic code. They didn't rebuild everything from scratch but literally used the code Elastic wrote.
No one argued OpenSearch is not OSS.
And sorry - if you think Amazon as a company is not screwing the small guy, I feel sorry for you.
> Also your arguments doesn't make any sense.
Thank you, I used your twisted logic to arrive at them.
> Lucene is still OSS and OpenSearch is not Lucene but built of Elastic code.
And Elastic is not Lucene but built of Lucene code. So why is it okay for Elastic to build a proprietary package using OSS code, but not okay for AWS to build an OSS package using OSS code?
> They didn't rebuild everything from scratch but literally used the code Elastic wrote.
1. OpenSearch is more than ES Community Edition. OpenSearch does have some code of its own; significant enough to matter. Code that was considered basic featureset that Elastic refused to add, even rejecting PRs. What do you do when your PRs that add significant features are rejected? Especially when the reason is, "because we don't want you to compete with our proprietary packages"? How FOSS is a project, really, when it refuses such PRs?
2. Like Elastic didn't rebuild everything from scratch but literally used the code (as a library) Lucene authors wrote. So why is it morally okay for Elastic to profit off of OSS work they didn't do, but not for AWS to build OSS off of OSS?
> No one argued OpenSearch is not OSS.
You called it 'stolen'. Why is it not stealing when Elastic builds a proprietary package off of OSS code, including code that was donated specifically to be included in an OSS package, when you claim it is 'stealing' to make an OSS package of the same license and with copyright notices preserved from another OSS package.
> Amazon as a company is not screwing the small guy
I know exactly where and how Amazon as a company is screwing the small guys. This is not one of them. This is a field so full of strawmen it is difficult to see where it ends.
> I feel sorry for you.
Please keep your sympathies to yourself. You need it more than I do right now. If nothing else, then for the simple task of judging your heroes with the same yardsticks as your villians.
I really don't fault Elastic for disliking the situation; someone making billions with you work while you're still counting coins sucks. But they reached their size due to OpenSource and AWS, so painting them as the profitless victim really does not show the whole picture. And their handling of the situation so far was horrible; they've even gone so far as to embed code in their client libraries so that they don't work with Amazons offering.
So yes, in conclusion, Amazon really has the moral high ground here.
That said I wish Elastic had chosen a better license and I wish they wouldn't charge for basic security options like SAML, RBAC, and encryption.
edit: And IIRC this all started because Amazon added authentication and RBAC to open source ELK and tried to submit the changes back to Elastic, which rejected them.
What better licenses or monetization strategy would have allowed them to charge enterprises who use it for more commercial purposes, and allow the individuals/casual users to use it for free?
This is exactly what Amazon wants you to think. That's why they're calling it "OpenSearch" and not "AmazonSearch".
And mind you, Elastic is no tiny startup. It has 2000+ employees.
Mind you this company basically is a monopoly in multiple domains. I’m afraid of the world where we align with the villain on something only because it’s convenient for us.
We're not. We're aligning with them because they happened to do something legitimately good. Being a villain doesn't mean that literally everything you do is bad.
What can make me certain I won’t ever need it? Ecosystems without monopolies tend to be more robust.
People often run this software on their compute, so obviously eliminating proprietary license complications makes that an easier option for their customers to deploy, and redirects some of that licensing money into the customer’s aws spend.
Don’t get me wrong, I’m happy there is a FOSS fork, but at the same time it feels like Amazon again has eaten another company’s lunch while feigning innocence.
If you are arguing that Amazon has no expectation of monetary reward, then I have a bridge I'd like to sell you!
But there's absolutely no way a company at the scale of Amazon could have got there without hiding some skeletons in the closet so it's good for the society in general to view ~~companies~~ Mega Corp. which has more money(read power) than several countries combined together always with suspicion.
They do deserve credit for stepping in to help even if it does support their own profit margins. Profit in itself isn't evil.
Not just calling them out, Even punishing them(fines) for their mistakes is not even a slap on their wrist at their scale we've seen this happening time and again with all Mega Corps.
There's just no incentive to not do something evil.
So extraordinary scrutiny for every single action seems to be the optimal approach until the governments decide to break Mega Corps. into smaller companies which can actually get hurt from punishments.
Can you give an example?
>> Even punishing them(fines) for their mistakes is not even a slap on their wrist at their scale we've seen this happening time and again
If this wasn't true then there wouldn't be sub-list for every wrongdoing spanning decades[1] i.e. If the punishments achieved it's purpose we wouldn't see continued offenses under same category.
Forcing employees to pee in bottles is pretty bad.
> If the punishments achieved it's purpose we wouldn't see continued offenses under same category.
Hah, the notion that punishments always work is funny. Good one. Try that on your kids. Look, making a punishment bigger may or may not help. I don't know about that. You can't just fine them out of existence for no reason, however. Bezos isn't Jack Ma and this isn't China.
Yes it is(Like dozens of other things listed in that Wiki), But I didn't ask that. I wanted to know how do you suggest we strike at 'the weakness of Amazon' to fix it (Like you suggested) that it treats it's employees with dignity and respect after that?
Because IMO if there's anything, It could have been done when Amazon's mistreatment of its workers surfaced way back in 2001.
>Hah, the notion that punishments always work is funny. Good one. Try that on your kids.
That's the point of punishments, isn't it? Just calling them out is convenient but does nothing.
> Look, making a punishment bigger may or may not help.
That's the whole point. They're too big for any punishments, Hence qualified people have been arguing that private companies shouldn't get that big(It undermines the basic tenants of democracy reg accumulation of power); But unfortunately those who can take action about it are also getting big thanks to $AMZN.
> however. Bezos isn't Jack Ma and this isn't China.
>> Calling the whole thing an evil empire isn't going to make any progress: your attacks are easily fended off.
You call them out for the individual wrongs like those listed on the Wikipedia page. Vote with your wallet. Debate and discuss.
> They're too big for any punishments
I don't agree with that. No company survives forever.
> Jack Ma & China
The US is not going to fine Amazon out of business like the CCP did to Jack Ma. The CCP is evil, not the people. Their crackdown on people's ability to freely debate and discuss just shoots the country's most valuable asset, its people, in its own foot over in over in order to maintain footholds for a relatively small group controlling the party. Any creative ideas are squashed. If the CCP would let up they would give rise to a new generation of ingenuity in China. Of course some CCP members might face criticism for that. It's up to them to decide whether or not to move in that direction. Nobody likes to be on the receiving end of criticism and you need to be strong to take it.
It's been happening at least since 2001, Starting with layoffs and now it has to come to -
> Forcing employees to pee in bottles is pretty bad.
> No company survives forever.
I agree, 'East India company' just survived for 273 years and people who ever affected by it are still suffering several generations later.
Also worth noting that other open-source search engines exist out there; it's entirely possible that other alternative technologies will become more attractive, depending on how all this plays out.
Amazon is the better steward of open sourced Elastic Search, hands down, because nobody else is doing it. I'm not even sure that stewardship of the technology's proliferation is TBD. Didn't Elastic close the source because they feared Amazon was taking too much of their potential revenue? Elastic gambled that only they could carry the technology forward, and they lost that bet. I'll grant it's possible Elastic could become more competitive with Amazon+OpenSearch somehow down the road, however it appears that right now Elastic is on its heals while Amazon is driving forward. It's also possible that another closed-source search engine becomes popular. That said, Elastic is built atop Lucene, and as long as an open sourced version of Elastic exists I'm not sure why you would start from scratch or with something else.
> Also worth noting that other open-source search engines exist out there; it's entirely possible that other alternative technologies will become more attractive, depending on how all this plays out.
Which open-source search engines do you think are competitive?
Perhaps - what are the commit/accepted contribution rates for the two projects like at the moment? (I'm not saying that's a perfect metric, but it could provide some information)
> Which open-source search engines do you think are competitive?
MeiliSearch[1] is an impressive-looking candidate. Perhaps two of the stronger innovations of Elasticsearch were horizontal scalability and the ability to pass documents to it over-the-network (basically, the ability to curl a JSON document and instantly have it be searchable -- and I'll admit/credit that Solr may have gotten there first with XML documents). Those two features make it easy to get started, and easy to scale up if needed (a common architect's concern).
> Elasticsearch is very powerful when it comes to massive datasets; it can quickly scale horizontally and allows one to build complex queries. The flipside of that is that nothing is easy. Even if all you need is simple search functionality that handles typos, synonyms, or filters, you're looking at hours of training, documentation, and configuration.
Spending some hours learning a new tool doesn't seem so bad to me when I'll be working with it as part of my product.
> For big tech companies dealing with enormous logs, Elasticsearch might make sense. But when it comes to medium-size datasets (i.e. less than five million rows) and smaller apps, it simply isn’t right for the job. Nonetheless, since no open-source alternative has historically been available, developers still go for Elastic as their default solution and wind up losing a disproportionate amount of time on setup and training.
> MeiliSearch is not made to search through billions of large text files or parse complex queries.
https://blog.meilisearch.com/why-should-you-use-meilisearch-...
I'm not sure I'd like to assume that my dataset will never grow. This is kind of like using training wheels. Some may find it helpful but I don't think it will grow to the widespread adoption Elastic has seen without a change of vision.
I work on an open source search engine called Typesense: https://typesense.org/
I'm biased, but I'd like to think that when it comes to site/app search, Typesense is quite competitive to Elasticsearch. Of course, ES has tons of features and configuration options, which is a blessing and a curse in terms of complexity and learning curve, so I wouldn't want to claim feature parity. But in terms of getting a good out-of-the-box search solution that works well for most use-cases and is easy to deploy and scale, I'm hoping for Typesense to be that go to.
I recently put this comparison matrix together: https://typesense.org/typesense-vs-algolia-vs-elasticsearch-...
https://mobile.twitter.com/ppgosavi/status/11791819009690378...
Amazon can show that EC2 predates the company by half a decade atleast
Granted, it looks like Elastic is trying to break compatibility pretty hard (see the new version checks in clients for example) but hopefully Amazon will release universal connectors, it certainly in the Amazon's interests.
Specifically, this feature https://github.com/elastic/elasticsearch/issues/42326#issuec... will be a big change for Elasticsearch. OpenSearch might try to mimic the API, but implementation details here will matter a lot, since this type of search is really picky when it comes to performance/recall balance. OpenDistro has already been working on their own version: https://opendistro.github.io/for-elasticsearch/features/knn.... ...so will they switch their API? Perhaps - but the results are going to be very different.
From the linked issue, it seemed that they were originally planning to develop this as a proprietary feature of Elasticsearch, without contributing the functionality to Apache Lucene, but then changed direction when the Apache Lucene developers (some of which are currently employed to do such work by Amazon) started to build its approximate nearest neighbor (ANN) vector search capabilities. [1]
It's great to see folks that work for Elastic collaborating and building on what is in Apache Lucene to extend the utility of ANN with Hierarchical Navigable Small World Graphs (HNSW) [2]! From this, I think it should be possible to implement an Open Source version of the functionality with a compatible API, if that is something that OpenSearch users seek.
Elasticsearch is still open source...just not in a license that Amazon can take advantage of.
But I don’t know how much of that is just ES and how much is Amazon
Last time we tried to upgrade our cluster version on AWS we ended up waiting for weeks until it randomly updated mid day on a busy day and broke everything.
It's a ton of small things that really make a difference.
T2 and T3s with no Master Nodes, let alone three, leads to a lot of problems.
When everything lives in MMAP'd memory and you are nickel and dime'n it with 4GB and one CPU. You're going to have a bad day.
The update was instant, what wasn't instant was the wait for the update. IDK if it was the maintenance window (which was set up for a weekend) or some bug, but we DID end up waiting weeks until it randomly updated.
They would have competed on the merits of the product. Looks like whoever is making these decisions is not familiar with what made them popular with Devs in the first place.
https://news.slashdot.org/story/21/08/07/0040229/elasticsear...
If they use GPL, it makes no difference as Amazon can still run it but if they use AGPL orgs would blacklist this because they don't want to be forced to open-source the rest of their code for their "elasticsearch integration"
True for me, I agree.
There were a lot of decisions and changes that were made that led to this happening, and we have no idea how it ends up.
I personally am interested to see what happens in the elasticsearch versus opensearch space. I want to see how this experiment plays out.
Just call it something new if definitions matter so much. "Fair Play Source", "Fair and Open", etc.
Fuck open source the way Amazon likes it. Look at how the giants abuse it to crush smaller innovators. It's not rewarding the original authors. It's turning them into labor slaves whose collective work is captured and monetized, with little hope of fair competition or reward.
Might as well call it "open slavery" and imagine Amazon as some Jabba the Hut creature.
The work was intended to be open. They just didn't imagine the giant Jabba slug eating the whole pie and freezing them out. Unintended, yet fatal, consequences.
The DOJ should look very closely at this behavior. It's anticompetitive af and wholly against the spirit and intent of the original authors.
I mean, I get why Elastic is not happy. But in this case, neither intent is pure, but at least the result on the Amazon side is.
Indeed. Amazon may be doing the right thing for the wrong reasons, but that's still better than doing the wrong thing for the wrong reasons, which is what Elastic did.