Hacker News Reader

Comment by giancarlostoro | Hacker News Reader

> Amazon's managed ElasticSearch offering is awful because it's ElasticSearch.

Had I never used ES or ELK I wouldnt even bat an eyelash. But man... This one hurt me in the tech feels. I already dont like ES when its on premise, I cant imagine on the cloud where you have even less control of it.

View on HN

I've hit basically every limit on CloudSearch when I was at a previous role. We increased the cluster count and index sizes to their 'absolute limits'. And we were still projected to blow through them by 2020.

I believe there's a project underway now to move it to ElasticSearch, but still, CloudSearch is and should always be considered a prototyping tool for proper ES implementations.

Is ES one of these necessary evils? I have been looking for a product similar to it, but it seems ES is always recommended.

I saw a post about this and the other open source alternative was solr.

Not sure about support/community though.

It's funny because under the hood, they're both Lucene.

I guess that's the power of open source, right?

No, ES is an affordable entry level search engine in a very expensive product space.

But there is nothing new in the standalone search engine space that I am aware of. Many engines have been gobbled up and integrated into larger product suites (see: Oracle Secure Enterprise Search).

I don’t find ES that bad, considering its power, but I haven’t had the best experience with the AWS service. Could be user error though....

I don't fault ES for this necessarily, but the biggest problem with ES is, it's easy to build something that looks like it works, right until it doesn't.

This is a problem for most DBs. You put stuff in, you take stuff out, and it works... but you're putting it in the DB in a way that will make the queries you need nearly impossible to do within your real-world/production constraints (response time, resources required, etc.)

The thing is ES is so different from most DBs, so you need to relearn how to do things right. And to make matters a little more difficult, all DBs have a lot of knobs, but ES just has way more knobs you need to know how to turn to get something that won't fall on it's face in production.

I don't remember my experience with ES on AWS that well besides it was behind on major versions for a while at one point, and if you set up ES resource with CloudFormation... god help you if you need to roll them back.

Using serverless with ES looked something like

api-1 api-2-because-1-is-frozen-rolling-back-cloudformation-es-instances api-3-because-2-is-also-frozen-rolling-back-cloudformation-es-instances api-4-...

Have you tried elasticcloud? They solve a lot of access problems present in AWS Elasticsearch along with providing the most recent version for deployment.

Elasticcloud has terrible security features - they removed IP whitelisting.

They do have user/password authentication though, which is something that barebones ElasticSearch doesn't normally have out of the box by default.

ES is one of those technologies that "everyone" goes to when they want any kind of basic text search indexing "at scale" (usually with a delusional notion of what their scaling needs actually are), because text search/indexing is tedious and hard at any scale that a basic DB query would fall over on (which is generally a much bigger scale than many want to admit).

The problem with ElasticSearch is that it does multiple things at once. It indexes documents. It stores documents. It searches indexes. It serves documents. The indexing and searching should be separated from the storage and serving at any kind of reasonable scale.

But, ES makes it “just work” at the proof of concept and low volume stages, and you can’t easily back out of it when you reach its limitations.

For all that people (rightfully) give them grief for being expensive, this is something Splunk got right. You can put your entire Splunk infrastructure on one host: like stem cells, a single Splunk box can perform any of the required functions (ingest, index, search). But you can also separate out each layer into separate hosts and then cluster them, so you have clusters of ingesters feeding clusters of indexers, accessed by clusters of search heads. That should be the architecture ES aims for, where people are given the flexibility to take whichever function is currently a pain point and break it out to a separate set of nodes.

Well, ES spent many years polishing itself, but there is one new very rough kid on the block:

https://github.com/toshi-search/Toshi/blob/master/README.md

Reply on news.ycombinator.com