We upgraded an old, 3PB large, Elasticsearch cluster without downtime
underthehood.meltwater.com
underthehood.meltwater.com
Luckily major updates from then have been relatively painless.
FogBugz was still on twelve ElasticSearch 1.6 nodes when I left in 2018. We also had a custom plugin (essentially requesting facets that weren't stored in ElasticSearch back from FogBugz), which was the main reason we hadn't spent much time thinking about upgrading it. To keep performance adequate, we scheduled cache flush operations that, even at the time, we knew were pants-on-head crazy to be doing in production. I can't remember if we were running 32-bit or 64-bit with Compressed OOPs.
Kiln was on an even older version, v1.4 if I remember correctly. And one of the shards had a corruption warning, yet it didn't seem to affect stability or results. But that wasn't a fun cluster to operate, since it refused to do certain types of maintenance because of the supposed corruption.
Hopefully the newer versions are easier to migrate between. I don't remember what exactly was preventing us from upgrading, but I'm sure part of it was wanting to avoid a full reindex.
Pulling off this level of scale with Elasticsearch is no easy feat and very impressive from a technical perspective. When you're running ES with petabytes of mission critical data as a core service powering the universe of a business, cluster rebuilds aren't an option (or maybe they are, as a last resort, but absolutely will not be acceptable on an ongoing basis).
Relying on Elasticsearch mega-clusters in this manner is akin to running an ultra-marathon with a really sharp pair of scissors glued in each hand. Or maybe even more extreme than my (admittedly lame) analogy.
Running nodes with such high shard counts is an appreciably precarious proposition, because there is a fair amount of overhead in the Elasticsearch management protocol. I wonder what the performance testing strategy entailed.
I have a lot of respect for the engineers working to make this project and service a success story. When it comes to Elasticsearch at scale, such outcomes are the exception.
As criticism, it's very vague, and as someone who doesn't work at Meltwater (for the last 5 years or so at least) it doesn't give me any information either. Well except that there are rumors about Meltwater, but that would be true about any large corporation.
Maybe I misunderstood and the horror stories were about ES, but I got it as being about the company itself. Could you expand? What type of stories? :)
And yes we have had our fair share of pain with the old cluster for sure.
the new version (7.17) is still behaving a lot better so far and feels a lot more predictable.
And during that time so many nodes had became slow and unresponsive that another (for us) previously unseen memory leak started to occur.
Nodes kept building up queues of unanswered ping requests on them. And the requests contained our 100Mb large cluster state, so the heaps filled up and evenmore nodes became unresponsive.
And from then on the whole thing turned into a death spiral of doom.
After trying, and failing to get it under control for 48 hours we gave up and rebuilt the whole cluster from scratch, using the snapshots we store on S3.
The recovery took another 90 hours or so. That was not a fun week.
One of the area managers - Kaveh, IIRC - also had a double morale on line with Trump. He was very "don't put your pen in the company ink", and proceeded to get one of his subordinates pregnant.
I remember there was an "anti Meltwater blog" at some point, but I can't find it now. I don't remember the URL either, so I can't look it up on archive.org. However, this site[0] seems to contain copy-pasted stuff from it.
As said, my experiences are 10+ years old, and hopefully things have improved.
I get that there’s little money to be made in these things but it’s surprising. Seems like most full text search are relatively simple plug-ins to existing databases or in memory only.
For example, until very recently Vespa did not even have case-sensitive string field matching [1]; doing a strict equality query on a string field was not possible, and the authors did not seem to see why it would be useful. Vespa lacks a lot of this kind of basic search functionality, making it less general-purpose than Elasticsearch.
It's clear that it's a very powerful search engine, though it also feels antiquated in many ways. It's very obvious that it's an ancient project that has been worked on by many different people throughout the years, with no cohesive vision or design, though it does seem like they're slowly cleaning things up. (The documentation used to be much worse, for one.)
Disclaimer: I was once part of it
> In order to control how queries are executed, we have built a plugin which exposes a set of custom query types. We use these query types to provide functionality and performance optimisations not available in stock Elasticsearch. For example, we have implemented wildcards within phrases, with support for executing within SpanNear queries. We optimise “*” to a match-all-query. And a whole lot of other things.
Did you port your the in-house plugins? Seems like a big blocker.
I don't want to spoil the other blog posts but we managed to solve almost all of our custom use cases without modifying elasticsearch itself. We still have one custom plugin but only to enhance functionality, not for performance and stability reasons.
Going up to i3en.12xlarge (or equivalent) would probably have worked as well.
But after that the impact of loosing just one node would be too big.