Lessons Learned from a Year of Elasticsearch in Production | Hacker News Reader

Lessons Learned from a Year of Elasticsearch in Production

tech.scrunch.com

115 pointskiyanwang19 comments

Not a bad article, but I wish they'd focus on search latency instead of thread pool size. Define your required 99.999th latency, draw a line on a chart, and tune/scale to reach it. If that requires SSDs, so be it. If you can reach your latency goals with huge thread pool spikes, fantastic.

Internal operational metrics like thread pool size are excellent for diagnosing failures to reach target latencies, but keeping thread pool queues low shouldn't be the direct goal.

Article author here. That's really good advice, thanks a tonne.

I used to try and reason with ES settings on production, for about a year or two. Something would invariably break, unless you spend time with it every day, checking configurations and monitoring performance.

We use the AWS ElasticSearch service, and while it does not provide the same flexibility, it's been fantastic for us. No configuration files, no weird restart behaviour, no thread pool madness - it just runs. You can upgrade your instance sizes seamlessly and it integrates well with other AWS parts (ex: cloudwatch).

How difficult would it be for the elasticsearch team to make the software run smoothly as a black box, i.e., without administrator supervision?

It's actually fairly difficult. There are so many different possible combinations of incoming data quality, quantity, system and disk configuration, performance and internal ES settings. Only an administrator would be familiar enough with the environment to monitor and tweak the configuration to make it the most performant.

There are a few hosted elasticsearch services available (Elastic Cloud, AWS ES, Qbox, etc) which would likely be a better option for the team that needed a hands-off elasticsearch cluster.

We chose found.io I figure since they write the code it better run smoothly for us.

If what you need is just search, you could try a SaaS, such as Algolia (https://www.algolia.com/)

+1. Algolia rocks. (https://www.spiralmedia.co.uk/implementing-lightning-fast-al...)

+1 Algolia too. Fast, simple, easy to tweak. It just works.

If you have a non-disclosure agreement with your clients, you cannot outsource any data processing / search. So in many cases a SaaS solution is a no-go, unfortunately.

Who'd pay them for support if everything ran smoothly?

People pay for RHEL, SUSE, Windows, Mongo, MySQL, etc.. support even though they may be part of a company perfectly capable of resolving any issue with the software. You get more than just basic support with all of these contracts. =)

> People pay for RHEL, SUSE, Windows, Mongo, MySQL, etc.. support even though they may be part of a company perfectly capable of resolving any issue with the software.

Regarding those examples:

1) For RHEL and Windows, paying is not optional 2) Mongo is so buggy most people need to pay for support in production 3) MySQL works well, but don't expect anything more than "forum answers" from Oracle support. Only 1 in 1000 companies paid for support according to the former president of MySQL AB.

Actually, companies are reluctant to pay for support for Open Source products, hence the business model tweaks of Redhat and the Open Core models of Nagios and other smaller Open Source vendors.

We went through various configurations/deployments of ES in-house, and eventually outsourced the job. The fickle nature of cluster restarts/upgrades were frustrating along with the unassigned & lost indexes that would often result. Then there were cluster partitions that wouldn't heal without a lot of manual intervention.

The tradeoff in value wasn't worth the time it was requiring, so we outsourced to compose.io -- they've been a good fit price & stability-wise, and feel pretty close to a blackbox for the benefit. If you need a large cluster however, you'll pay handsomely with any of the SaaS providers when last I looked at it. For large quantity work we've moved to AWS Redshift.

We've run it without supervision for about a year now at multiple sites, and have not had any major issues.

Any chance you could provide configs and/or insight for the other people here who have problems?

Our indexes are probably very small (about 40gb total) but I've found if you value speed and consistency over all else.... just run a single, 1 shard node and have a fallback in case of random errors. I just do major version upgrades and rebuild the indexes, no real maintenance.

I should add that all "hot" data fits in 24g of memory.

I wonder what size of cluster they are using.

I have two clusters running.

Largest one is an ELK stack. 7 nodes (soon to be 8, we're logging too much :) ), no dedicated master nodes, ~3TBs of data (1.5TB data x 2 for the back up). Need to make sure they are split appropriately into availability zones in case you lose a cloud zone. Need to have the right # of minimum master nodes otherwise you can hit a split brain issue. 500GB SSD, 4vCPUs, 26gigs memory. It eats ~5 gigs an hour peak load without breaking a sweat. Can basically take any heavy query people have thrown at it kibana wise.

As a counterpoint, I to run ElasticSearch in production.

The cluster is much smaller than others---having 12 nodes eating 32GB memory a piece, and handling only 50GB of data. I run a consumer-facing website from it, where all queries hit the ES cluster with limited caching at a pace of ~8,000 qps at peak for fairly heavy computation aggregations.

ES scaling has consistently been simple as adding another box ever 3-6 months.

Reply on news.ycombinator.com