Hacker News Reader

Comment by jdc0589 | Hacker News Reader

> Elasticsearch supports automatic shard rebalancing, which would let us add new nodes to the cluster, fulfilling the linearly scalable requirement out of the box.

I'm assuming they mean "I can add a new node to the cluster, and NEW SHARDS FROM NEW INDEXES can be distributed amongst the new nodes".... So far as I know elasticsearch can't rebalance shard location or composition automatically based on cluster membership change events...right? I mean, the cluster reroute api will let you manually move a shard, but thats all I know of.

View on HN

ES can indeed rebalance shards to new nodes as they join the cluster. More here: https://www.elastic.co/guide/en/elasticsearch/reference/curr...

huh, the more you know. not sure how I've missed this. I'll definitely look for some discussions/use cases of how this actually plays out.

I suspect you were thinking of changing the number of shards, which can be an important part of rebalancing an index. Unfortunately, Elasticsearch can't do that.

There is a convenient reindex API available in more recent versions, although in most cases where you can rebuild your index anyway, it's faster to do that.

Ahh. I was wondering when they were going to get around to implementing that. I might have to upgrade to 5.x soon.

FYI, the reindex API was added as experimental in 2.3, so it's been there for about a year [1]. Even as experimental it worked great.

[1] https://www.elastic.co/guide/en/elasticsearch/reference/2.3/...

Ha. I just came here to edit my comment. I remember seeing it before and just checked the docs again. In 5.2 it's STILL considered experiential. Craziness. It took me about an hour to build The basic functionality myself.

Thank you for noticing this, it is not considered experimental anymore. I integrated a change[1] to update the docs; it should percolate through to the site within thirty minutes.

[1]: https://github.com/elastic/elasticsearch/pull/23621

FYI, the re-index API is awesome.

Great work. It appears to have already percolated.

Tuning cluster rebalance settings can be... challenging. I've done some bad, bad things with a cluster where it would effectively balance things back and forth - forever.

Fiddling with cluster.routing.allocation.balance.* can really paint you into a corner quickly.

Yes that's actually one of my favorite ES features. Kafka doesn't do any type of automatic topic partition rebalancing which was quite shocking to me when I first started using it. Seems kind of primitive in comparison.

Reply on news.ycombinator.com