Comment by jillesvangurp | Hacker News Reader

Comment by jillesvangurp | Hacker News Reader

Interesting overview.

I understand from elsewhere on their site that they are not using a recent version of ES (2.x?). This has a big impact on how things behave as they improved performance and garbage collect behavior by e.g. doing more things off heap with memory mapped files, fixing tons of issues with clustering, refactoring the on disk storage in Lucene 5 and 6, and giving users more control over sharding and indicex creation. So, it's likely that this setup is working around issues that probably have been addressed in later versions.

For example, 6.x has a shard allocation API that allows you control which nodes host which shards. That sounds like that would be useful in this setup.

They also seem to be mixing query and indexing traffic on the same nodes. I would suggest separating these in a cluster this big and using specialized nodes for query and index nodes (as well as dedicated master nodes). That way write node performance would be a lot more predictable and query traffic is no longer able to disrupt writes. Also you can use instances with more CPU/memory for querying or use elastic scaling groups to add querying capacity when needed. You might even get away with using cheaper T instances for this with cpu credits.

Hi. We are actually using an even older version, 1.7.6 for our cluster. This is off not an ideal set-up and we will upgrade as soon as we can. That is not an easy task in a cluster of this size and given our uptime requirements. We know that general search and indexing performance, shard allocation and general cluster state management has all improved a lot in later versions of elasticsearch but we still anticipate that we will need shardonnay running even after we upgrade.

The shard allocation api would also be a very helpful addition to this solution. Currently we can sometimes get into situations where elasticsearch actively works against shardonnay and does very 'stupid' moves out of our control that later needs to be 'fixed' by shardonnay in later iterations. We have tried to disable elasticsearchs own allocation algorithm as much as we can but we still want it active as well as a safety net in case something unexpected happens with the hardware for example.

We do have dedicated master nodes. We also will have query and indexing client nodes that coordinates and merges search responses and handles routing etc. But behind that we utilize all data nodes for both query and indexing load. That has worked well so far due to the fact that indexing load is fairly constant. And we scale query load on recent indexes by having a large number of replicas for those.

Our largest problems have always been the vast difference in query complexity that we get. But we are slowly getting on top of that problem now as well. Some details on how that is done can be found in the following blog post

https://underthehood.meltwater.com/blog/2018/09/28/using-mac...

I fully sympathize, I also still have a 1.7.x cluster that I have wanted to get rid off for years now. Several rounds of compatibility breaking changes have made this hard for us. Impressive that you can make this work at this scale.

The thing is, at the scale you are running that probably means that you are running at a much higher cost than technically needed with a recent 6.x setup. You should be seeing magnitudes improvement in memory usage probably.

Recent versions of ES are much better at preventing complex queries from taking down nodes. They've done a lot of good work with circuit breakers preventing queries from getting out of hand. They are also a lot smarter at e.g. doing caching for filtered queries.

My guess is you should not need Shardonnay on a recent setup. Most of what it does should be supported natively these days. ES has been working with many companies on setups comparable to yours and they've learned a lot in the last few years how to do this properly.

Another feature that could be of interest to you is cross cluster search introduced in 6.x (replacing the tribe functionality they had in v2 and v5). This would allow you to isolate old data to a separate cluster optimized for reads. Probably whenever you hit those old indices, your query complexity goes through the roof because it needs to open more shards.

Honestly, I bet 6.x doesn't work well at PB scale. There may be different problems with different solutions, but I bet it would still need custom shard balancing, etc. I work at TB scale and 6.x largely works well with the exception of reindexing. Reindexing without downtime is still tricky.

The reason I think there are likely still problems at PB scale because of the attitude of ES core developers. They collectively act as if reindexing is no big deal and their proposed solution to many things I consider bugs is "just reindex". Reindex is the last thing I want to do given it is so hard to do it at TB scale with zero downtime. I don't think the core developers have experience with large clusters themselves so I find it unlikely that just upgrading to 6.x would solve all the problems at PB scale.

I know people running ES at PB scale. You definitely need to know what you are doing of course but it is entirely possible. There are definitely companies doing this. Any operations at this scale need planning. So, I don't think you are being entirely fair here.

I'm not saying upgrading will solve all your problem but I sure know of a lot of problems I've had with 1.7 that are much less of a problem or not a problem at all with 6.x.

Reindexing is indeed time consuming. However, if you engineer your cluster properly, you should be able to do it without downtime. For example, I've used index aliases in the past to manage this. Reindex in the background, swap over to the new index atomically when ready. Maybe don't reindex all your indices at once. Also they have a new reindex API in es 6 to support this. At PB scale of course this is going to take time. I've also considered doing re-indexing on a separate cluster and using e.g. the snapshot API to restore the end result.

In our case at Meltwater we also have new documents as well as updates to old documents coming in. Meaning older indexes are actively being modified with low latency requirements on visibility and consistency of those modifications. This makes it more tricky to both reindex as well as doing a seamless/no-downtime-whatsoever upgrade to a new major version. It's not infeasible at all though, and we're working on it. Reindexing a PB of data should be possible on a matter of weeks based on our estimates, but we shall see!

We’re still running 1.6 and 0.90 for a couple production workloads. Thankfully the 0.90 cluster requires no maintenance, because pre-1.0 things are pretty different.

We at Meltwater have some custom plugins for Elasticsearch that make a number of modifications to how queries are executed, and completely replace some low level Elasticsearch query implementations. We're also running a custom ES 1.7 version with some features backported from version 2+. The end result is something like 5-10x lower GC pressure and radically increased performance and query throughput for our particular workload. Without these changes we'd not be able to sustain our workload without a massive amount of more hardware and cost, just like you say.

Our flavor of Elasticsearch 1.7 is faster than vanilla 2.* for our workload, though still slower than ES 2.* with our customizations applied.

Recent Elasticsearch versions still use the same basic shard allocation algorithms as far as we know. Our workload is very imbalanced towards recent data, but it's not a binary hot-cold matter, rather a more exponential decay in workload for older indexes. We fully expect to need Shardonnay to balance the workload even with ES 7+.

We're also in early conversations with Elastic about shard placement optimization. They seem to be interested in applying linear optimization in a similar way, with a goal of solving the fundamental problems with shard allocation based on observed workload.

Reply on news.ycombinator.com