Hacker News Reader

Comment by ordiel | Hacker News Reader

When people say "Lightweight alternative" most of the time they are implying less features also, they are not lightweight just because

View on HN

I would say a fraction of the features. If you don't need those features, that's fine of course. But in my experience, I end up using a lot of those features when dealing with real world customer requirements. I guess if you implement search for some website where search quality just doesn't matter, light weight is fine. For everything else, using proper solutions might be the wiser thing.

In terms of performance you see a lot of comments that are stating X is faster than Y. I'd take such comments with a large grain of salt. Unless those lightweight alternatives actually do the same things you can't really compare the performance. It's not that hard to make indexing fast in Elasticsearch if you disable all the features that make it slower. Of course if you don't actually have those features it's going to be fast by default because it simply isn't doing anywhere close to the same things.

Elasticsearch is actually pretty damn fast even when you do use its many features. The reason is that it relies on in memory caches, thread pools, etc. and that a lot of very smart people have been implementing and optimizing very efficient algorithms in Lucene for the last 25 years. Elasticsearch actually can run with as little as 256MB but of course you are not going to be able to cache a lot of data with that and performance will suffer accordingly. Mostly large heap sizes with Elasticsearch are all about using larger caches. It also relies on memory mapped files and OS file caches for that. That's what allows it to work with extremely large data sets and still provide query responses in milliseconds.

There's no reason you wouldn't be able to do the same with a native implementation of course. But it would be naive to assume you'd end up using a lot less memory or CPU. Using that would be pretty core to matching performance and features. Not using that would make it pretty hard to get even close.

Yeah. There's a lot that Elasticsearch can do. There are libraries and applications that can do bits of what ES can do, but no-one I'm aware of (and I watch this space) is even slightly close to building a genuine replacement (I'm not counting the AWS fork, for obvious reasons)

How much sense does it make for another product to try to be even slightly close to elasticsearch?

My experience with ES has been somewhat mixed, exactly because them trying to do everything at the same time and a lot of the specific things we needed being under documented and involving much more try and error than should be necessary.

I for one welcome more focused products, when the need again arises for something in this space.

I see search engine space similar to the relational database space… more features is really a good thing it helps you answer more questions about your data … just my 2 cents

Well, I suppose anything could be a genuine replacement depending on how many of the features of elasticsearch the user actually needs/uses.

However ... unfortunately, you can't just drop features from ES (or most software really) to make it lightweight.

Java already does this AFAIK. If you use a GC that supports class unloading (all the new ones do).

Unused classes can be unloaded from RAM and it's like they don't exist. The executable is still huge but lower memory and CPU cache footprint.

For a lot of use cases, you can.

Where would you say Solr falls short? In terms of search - it's definitely the case that ES has expanded beyond that (metrics, apm...)

I feel ES has been trying really hard to walk away from the features Solr is good at. While ES still supports multiple languages and custom tokenization chains and even custom pre-processing chains (somewhat equivalent to Solr's UpdateRequestProcessors), I felt that they were very deeply buried in the configuration, when I look at ES a year ago.

ES is truly focusing on metrics and things and does have some features to make those use cases easier that Solr would probably need a lot of configuration/customization for.

So, Solr is about search. ES is about a specific set of use cases that rely very heavily on search.

And Lucidworks Fusion (commercial alternative to ES) is about big data and ML and full multi-tool pipeline on top of Solr.

Elastic makes the bulk of their money from log search. Developer productivity tools like Splunk as main competitor.

So that’s where they’ve invested a lot in tooling and visualization.

ES wins hands down in search in my experience because the companies that were doing search stuff "before it was cool" were mostly running stuff like Autonomy DRE which work well but are expensive and proprietary.

There is a huge market just by going after that kind of customers.

They both use Doug Cutting’s lucene under the hood

Except every person has a different set of 10% features.

but its written in Go

Reply on news.ycombinator.com