Thoughts on Algolia vs. Solr and Elasticsearch | Hacker News Reader

Thoughts on Algolia vs. Solr and Elasticsearch

opensourceconnections.com

101 pointssoftwaredoug32 comments

I remember a previous employer asked me to talk to one of the authors because I have NLP and search engine design experience (I forgot which author). I remember he kept saying they don't do big data and that most NLP stuff other search engines use are irrelevant because their product works with the type of search they do. I asked a couple of complex questions which they disregarded as not important for their product.

This was probably 3 years ago or less so the product might have changed, but what I got out of that chat was that Algolia is for websites that want to add a search functionality with a nice UI without much hassle.

But if you are doing something complex it's does not compare to Solr or Elasticsearch

Again this was a while ago.

My experience has been that if search isn't critical to the core of your business, you should use Algolia. What I mean is that good search is hard and expensive to build, and if your product will not suffer from good enough search then it will be an order of magnitude cheaper. I use it to search over a list of 25k ish items that can be ordered broken into 4 categories. The users are domain experts and know what they're searching for. Algolia took me 45 minutes to set up and it returns results almost as fast as I can type, it was a no brainer.

However, if I had users that needed to lean on search to find things they didn't know about I would want to look into something that could infer meaning from the text input.

(Author here) Case in point where Elasticsearch shines is one of yesterday's hacker news articles

http://sujitpal.blogspot.com/2016/05/elasticsearch-based-ima...

Here Elasticsearch is used as a framework where "search" really means some kind of distributed feature similarity. But even text search where your incorporating a lot of data science, external signals, or semantic awareness can fall more into the category of distributed feature similarity.

Algolias strength is in a simpler path to straightforward and easily understood search. I'm not sure it's the path to amazing and deeply customized search (or search-driven features). Algolia gets far more right out of the box at a configuration level many need when they can't put a team around building an amazing Solr/ES experience.

My hope is Solr/ES can learn a thing or two in the ease of use dept with relevance!

If you like this article don't forget to listen to us chat with Algolia CTO on our podcast Search Disco

http://opensourceconnections.com/blog/2016/05/31/algolia-sea...

It's like the liner notes for the blog article...

I've had great experiences with algolia. I've used it on numerous client projects, and my own product https://appapp.io .

It is great as an out of the box search solution that will probably fit 90% of cases. It is blindingly fast and my clients love it.

I expect there are some cases where elastic or solr will be a better fit; but these are most definitely edge cases for very specific search requirements. Algolia is a killer app for generic 'good' search.

In my experience, Algolia is OK for toy sites but not for anything even moderately complex. Even on HN it's often a challenge to find the right results. They seem to have gone with the philosophy that speed is more important than relevance. Keyword matching is only one small aspect of a good search experience.

For me Medium, Product Hunt, Digital Ocean or even Twitch are not "toy sites".

Haven't tried the others, but Product Hunt's search is also extremely poor. Just today I searched for "mac voice control" (unquoted) and besides being very sluggish because of all the refreshing it does while I type, it returned 0 results. What I was looking for was Lacona [1] which was just featured a couple days ago and has a description containing two of the words from my query (1/3 of the description!).

[1] https://www.producthunt.com/tech/lacona

This is actually a great example that shows where Algolia doesn't shine.

Algolia is great a known-item searches. Searches that are like looking up a contact in your phone list. This is a common use case with search, but one of only many.

Your use case is closer to grasping at straws because you don't know the language to use. You can't quite remember the name for something. This is also very common, and not handled well with purely string matching. Another example is in this blog article[1] where I talk about searching for "manual lawn mower" not realizing the right terminology is a "reel mower." Mapping vernacular is a hard problem, and I feel like Solr/ES while harder are better equipped for the problem.

(I also often find instant search distracting)

[1]http://opensourceconnections.com/blog/2014/06/10/what-is-sea...

> Algolia is great at known-item searches

I have not used Algolia much but that's not a particularly compelling selling point, most search engines tend to get it right if you know exactly what to query. The problem is this is, like, 1% of the difficulty of making a good search engine.

Totally agree, but in my example above I'm actually not even requiring vernacular mapping or any sort of conceptual disambiguation. Basic TF-IDF should rank that result pretty well and it certainly should not return 0 results.

Great article by the way!

@Lxr

In addition to being good at known item searches, my pitch for Algolia would be

(1) it's Easier to perform relevance tuning. You'd be surprised how annoying the Solr/ES defaults are. And the API is more advanced. (2) it has much better typo tolerance (again this gets at "known item" where you fat finger something)

Algolia contains a lot of customization parameters that can be used. For example the Product Hunt example could have been solve easily by using removeWordsIfNoResults=allOptional (query is trying as a AND and if there is no results, it is tried again as a OR)

Searching google for "producthunt mac voice control", the desired result came at number 4. Not bad.

Ha! First result for this:

site:producthunt.com mac voice control

This result is indeed not good but it could be solved easily with one Algolia setting (removeWordsIfNoResult=allOptional which perform the query with all terms as mandatory and reply it with optional terms if there is no result).

Algolia comes with a lot of pre-defined tuning that are good for most use cases (see https://blog.algolia.com/inside-the-algolia-engine-part-3-qu... for more details).

That said there is always some tuning to have perfect result for a specific use case. There is no engine that provide perfect results out of the box without any tuning.

True though this assumes these direct words are mentioned in the text (assuming know synonyms). Algolia has basic synonym functionality but ES gives you a lot more power here.

Further the more words you remove, the less constrained the results get possibly creating a lot of noise. Though I assume you remove based on document frequency, which helps.

This would be a problem with this strategy regardless of search engine (ES or Algolia)

Algolia has actually a very decent synonyms support:

  * mono & multi-words synonyms,

  * typo-tolerance is compliant with synonyms,

  * matching synonyms are highlighted,

  * prefix search works also on synonyms (even on multi words).

A v2 of the synonyms API will be released in the next few days including:

  * dedicated API endpoints (leveraging existing synonyms but also Algolia's "alternative corrections" and "placeholders")

  * new edition UI (with search & filtering capabilities)

  * one/bi direction synonyms

nice I stand corrected. The first list is what I knew about!

This is actually an interesting dividing line. As you get more sophisticated modeling fuzzier concepts over just synonyms (synonyms are just one tool for this) then I think you get out of algolias sweet spot. When text pushed into a feature space (which happens quite a lot when you're mapping vernaculars) you begin to gain more from Solr and ESs depth of customizability.

> Algolia is OK for toy sites

Orthogonally agree. I can usually tell when a site's running Algolia by the asymmetry between the quality of the results (useless) and the familiarity of the site (high). It feels like the product is endorsed more for social, versus technical, reasons.

We actually have pretty complex implementation done with Algolia. Like a grocery store where relevance depends of the history of each user.

As with any engine, there is a lot of customization possible to handle complex use cases and it won't be handled out of the box.

You can read this list of 10 tips to have a great relevance, all of them are doable with Algolia: https://blog.algolia.com/algolias-top-10-tips-to-achieve-gre...

I recently had to build a product search functionality. I was able to pick up nearly all important features of Elasticsearch in about 2 days and had live-indexing and search up and running within 2 more days. Of course, this wasn't production level scalable code. But it was fairly easy to hook a Logical Decoding Output Plugin on Postgres, which would stream database mutations (in form of JSON) to a Kafka cluster, from where the Elasticsearch layer will ingest the data and index/update it appropriately.

what logical decoder?

Postgres 9.4+ lets you use the write-ahead log and "replication slots" using which you can create streams of database changes that can be replayed to a client in the order they were made on the origin server [1]

EDIT: In case the question was which library did I use, I used BottledWater and it's worked great for me so far [2]

[1] https://www.postgresql.org/docs/9.4/static/logicaldecoding-e...

[2] https://github.com/confluentinc/bottledwater-pg

One of the least talked about search engines is NodeChef Cloud Search. Anybody using any language could use it insofar as they have a mongodb driver. Anyone who used it could comment on it? https://nodechef.com/nodechef-search-and-sql-analytics

Whats your referral code?

Of course, if you like what Doug has to say and are building your own search functionality on top of Solr or Elasticsearch, you'll want to buy his book:

https://www.manning.com/books/relevant-search

Why would one use Algolia over Postgres's built in text search features? Ex: http://rachbelaid.com/postgres-full-text-search-is-good-enou...

I already have Postgres search queries running on the order of 1-3ms for large queries, basically faster than Algolia, with seemingly the same feature sets.

I don't understand the value proposition over something like Postgres, for sites that already have Postgres databases. (And I'm sure other databases might offer similar built-in search capabilities, I just don't know those other RDBMSs)

> Why would one use Algolia over Postgres's built in text search features? Ex: http://rachbelaid.com/postgres-full-text-search-is-good-enou....

Postgres built-in search features are really for large document searching -- we've tried using it for indexing e.g. an array of tags for auto-complete and the search times were in the ~500ms+ range, as a matter of fact the non-indexed regex search was only slightly slower. I thought the later was rather surprising - but asking on the mailing-lists, I was told that this was basically working as intended.

edit: I don't fully understand the value-proposition of Algolia either, most of the trouble with going outside of your production database for search - is the problem of keeping your data in sync and current with the production database; compared to that setting up an Elasticsearch cluster really isn't such a big deal, there are also half-way solutions, for example Amazon has an Elasticsearch-service.

I would recommend to read this post to have an idea: https://blog.algolia.com/inside-the-algolia-engine-part-3-qu...

Textual relevance is a very complex domain and the Postgres's built in text search features is a simple keyword matching engine compared to Algolia engine that contains a lot of alternative matching. The mesure of textual relevance is also very different of what you have in the Postgres text search.

At the end, this is not only about speed but mainly about relevance

If Postgres works for you then great, but it doesn't really compare to Solr/Elasticsearch. I haven't used Algolia so I cannot comment on it. Postgres is hard to use beyond very basic keyword searching and it doesn't support many standard features such as phrase search.

Interestingly, Solr also can do a lot of database type work. It can store not just documents for text searches, but also for numeric fields, aggregations, pivoting, etc. A nice tool for a data analysis back end. Even the Hue project uses it for its dashboard maker.

Reply on news.ycombinator.com