karlney

Shameless plug, but one of the other blog posts in the above series explain how we addressed high and unpredictable memory load in relation to wildcard searches.

https://underthehood.meltwater.com/blog/2022/11/25/how-we-up...

After those changes were made then we haven't had any major issues with heap spikes. But it depends on a lot of factors, how your data looks, how your queries look and what aggregations that are made so its very hard to give some always applicable advice.

This is a good resource for the up to date resilience status in Elasticsearch https://www.elastic.co/guide/en/elasticsearch/resiliency/cur...

They have gotten A LOT better since the 1.x days

We feel that ~300 nodes strikes a good balance in the cattle vs pets philosophy.

Going up to i3en.12xlarge (or equivalent) would probably have worked as well.

But after that the impact of loosing just one node would be too big.

One time, a few years ago a particularly nasty query was executed over and over again and it took a few hours to find it and then block it.

And during that time so many nodes had became slow and unresponsive that another (for us) previously unseen memory leak started to occur.

Nodes kept building up queues of unanswered ping requests on them. And the requests contained our 100Mb large cluster state, so the heaps filled up and evenmore nodes became unresponsive.

And from then on the whole thing turned into a death spiral of doom.

After trying, and failing to get it under control for 48 hours we gave up and rebuilt the whole cluster from scratch, using the snapshots we store on S3.

The recovery took another 90 hours or so. That was not a fun week.

Thank you for those kind words.

And yes we have had our fair share of pain with the old cluster for sure.

the new version (7.17) is still behaving a lot better so far and feels a lot more predictable.

Thank you. Yes it was a massive project.

I don't want to spoil the other blog posts but we managed to solve almost all of our custom use cases without modifying elasticsearch itself. We still have one custom plugin but only to enhance functionality, not for performance and stability reasons.

Thanks for this info!

Doing a quick googling on elasticsearch and ZGC I just found this https://github.com/elastic/elasticsearch/issues/44321 where the official response from elastic is that only G1GC and CMS are tested and supported.

I still think that it will be worth trying Shenandoah or ZGC if/when we ever get down to do some experiments with larger heaps.

Hi, do you have real use experience running elasticsearch with 64g+ heap?

Is there any articles/benchmark/notes or anything that you would be willing to share?

We have considered trying out 64g+ heaps for our cluster but we are concerned about very long gc pauses impacting the search performance.

The social data are fairly small documents, but we index lots of other things as well that is larger in size. Another factor is our large replica count, especially for the hot data.

We also have quite a lot of metadata.

We use datadog for monitoring our cluster. Shardonnay also produces lots of custom metrics that you don't get from the vanilla elasticsearch datadog integration.

We tried the hot/cold architecture as well, and used it before in our data centre architecture.

But currently we have concluded that it is not worth the added complexity. But that might change again if/when we learn more or get new requirements.

We do change the number of replicas for recent data vs old data. We actually have a 4 tiered approach to how many replicas we use.

Hi. We are actually using an even older version, 1.7.6 for our cluster. This is off not an ideal set-up and we will upgrade as soon as we can. That is not an easy task in a cluster of this size and given our uptime requirements. We know that general search and indexing performance, shard allocation and general cluster state management has all improved a lot in later versions of elasticsearch but we still anticipate that we will need shardonnay running even after we upgrade.

The shard allocation api would also be a very helpful addition to this solution. Currently we can sometimes get into situations where elasticsearch actively works against shardonnay and does very 'stupid' moves out of our control that later needs to be 'fixed' by shardonnay in later iterations. We have tried to disable elasticsearchs own allocation algorithm as much as we can but we still want it active as well as a safety net in case something unexpected happens with the hardware for example.

We do have dedicated master nodes. We also will have query and indexing client nodes that coordinates and merges search responses and handles routing etc. But behind that we utilize all data nodes for both query and indexing load. That has worked well so far due to the fact that indexing load is fairly constant. And we scale query load on recent indexes by having a large number of replicas for those.

Our largest problems have always been the vast difference in query complexity that we get. But we are slowly getting on top of that problem now as well. Some details on how that is done can be found in the following blog post

https://underthehood.meltwater.com/blog/2018/09/28/using-mac...