Hacker News Reader

Some free advice: reduce the number of shards! Each shard comes with some state management overhead.

The soft limit is currently at 1000 shards per node, but you should be aiming at 25-50GB of data per shard.

How did ES 6 reduce your operation costs by 50%? Same CPU, disk & network bandwidth?

That's an insane number of shards, you should be closer to 500 for that amount of data and only 6 nodes.

It's due to the number of indicies/indexes stored from our various data sources. Yours and another poster's comments are interesting so we might look at ways we can reduce the shard count given the new info on overhead.