Typesense: Open-Source Alternative to Algolia
github.com
github.com
Both MeiliSearch and Typesense use a reverse index with a Levenshtein automaton to handle typos, but when it comes to sorting document:
- Typesense use a default_sorting_field on each document, it means that before indexing your documents you need to compute a relevancy score for typesense to be able to sort them based on your needs (https://typesense.org/docs/0.11.1/guide/#ranking-relevance)
- On the other hand MeiliSearch, uses a bucket sort which means that there is a default relevancy algorithm based on the proximity of words in the documents, the fields in which the words are found and the number of typos (https://docs.meilisearch.com/guides/advanced_guides/ranking....). And you can still add you own custom rules if you want to alter the default search behavior.
This is not entirely true. You can just use a field with a constant value (say 100) as the default sorting field and Typesense will just use the text based relevancy. Please update your comment.
The reason why Typesense does insist on one, though is that it's always a good idea to have a field that indicates popularity of a record (or a proxy to it). It makes search so much better.
(those two are a bit like Lucene + ElasticSearch — but written in Rust)
Lucene and Lucene-like projects (Tantivy or Bleve in Golang) are general-purpose search libraries. They can handle enormous datasets, and you can make very complex queries on them (compute the average age of people named Karl in a certain type of document for example).
These libraries are based on tf-idf (term frequency inverse document frequency) algorithm and manage quite poorly typos for example (unless you make the setup to index your documents differently to parse them correctly).
Toshi is like Elastic for Lucene, it provides sharding and JSON over HTTP api.
You can basically used Lucene and its derivatives for basically any search related project, but you may have to dive into how it works and understand concepts like tokenization or ngrams to tune it according to your needs.
On the other hand, MeiliSearch (and I guess Typesense, but I can not talk for them) focus a subset of what you could build with Lucene or Elastic.
It is a fully functionnal Restful API, made for instant search or search-as-you-type. The algorithms behind MeiliSearch are simply different: a inverse index, with a levensthein automaton to handle typos, then a bucket sort you can tune for the ranking of the returned documents. The aim is to provide a easyer go-to solution to implement for customer-facing search.
You won't be able to make super complex queries on terabytes of data. We just make super fast and ultra relevant search for end-user.
TypeSense and MeiliSearch focus on the same usage, we choose Rust for performance, security and the modern ecosystem that will allow easier maintenance :D
@fulmicoton and @tpayet, I'm thinking then if I want both full text search, and also faceted search, then Tantivy can do that, but at this time, MeiliSearch (and Typesense) don't do that?
( When I look here: https://github.com/meilisearch/MeiliSearch in the features list, I see no mentioning of faceted search. Whilst Tantivy does list faceted search as a feature: https://github.com/tantivy-search/tantivy )
@tpayet, for a database like MeiliSearch, is faceted search typically always off-topic? Or you're thinking about adding faceted search, later on?
(My use case is 1) full text search, 2) typo friendly, and with 3) e.g. "begin" matching also "began", "begun", and "run" also matching "running", and 4) in all lanugages, and 5) faceted-search restricted to tags and categories and user groups.)
MeiliSearch does not offer faceted search yet. It is one of the key feature that we are still missing but we plan to work on it in the coming weeks.
For your use case today, I suggest you use Typesense if it fits your needs ( they handle faceted search already ) or Tantivy or Toshi.
To manage different languages, you should make one index per language. MeiliSearch and Tantivy handles kanjis! We will add faceted search in the coming weeks if you are not in a hurry :D
As of now, my project uses ElasticSearch — it works fine, but it wants lots of RAM which I find slightly annoying, ... and the new Java v 9, 10, 11 etc feels a bit worrying. — So, as of now I'm just staying up-to-date with new Rust based search engines.
based on http://highscalability.com/blog/2013/1/28/duckduckgo-archite...:
The fat tail queries go against PostgreSQL and the long tail queries go against Solr. For shorter queries PostgreSQL takes precedence. Long tail fills in Instance Answers where nothing else catches.
It seems that Bing is now a part of their sources indeed: https://help.duckduckgo.com/results/sources
Sonic is "just" a inverted index with a levenshtein automaton, it will returns only the documents ids in which there is your requests words and then you will have to retrieve the full documents in your main database and only then can you apply some relevancy ranking by yourself.
I have played around with lucene a lot and it seems typesense is a very close match to the feature set. - Apart from the REST interface on top.
Was the decision to not use the mature lucene platform technical? The memory and hardware requirements of lucene are quite small, even if Elastic or Solr leave a very different impression.
Glad to see a solution positioning itself a bit leaner than Solr/Elastic though, they really are a bit heavy for many occasions.
Most of the tricks that Algolia and (presumably) this product do to be fast have more to do with how they manage memory than with their implementation language. Basically, Algolia is a non sharded search index that is loaded into memory in its entirety. They don't have to worry about disk seeks, file cache misses, etc. Not a problem when your entire index fits in memory. Of course that puts an upper limit on what they can index and limits the uses to smallish datasets that in no way pose any challenge whatsoever to Lucene (especially when your index easily fits in memory).
I'm not saying it's a bad approach. I think it's a great approach for smallish data sets with very loose ranking requirements where a one size fits all solution does the job and is good enough. For many companies search is not really core to their experience and it just needs to be idiot proof and low hassle to set up for their non expert tech teams and product managers.
> when 1 million Hacker News titles are indexed along with their points, Typesense consumes 165 MB of memory. The same size of that data on disk in JSON format is 88 MB.
I like the compact filter_by, sort_by with qualifiers
let searchParameters = { 'q' : 'harry', 'query_by' : 'title', 'filter_by' : 'publication_year:<1998', 'sort_by' : 'publication_year:desc' }
Storing everything in memory seems fast, but seems like it'd be quite the resource hog on a server -- is that a normal approach to take?
It's reassuring that the examples and documentation all revolve around books (as my data set is actually ~55 million books also), but since theirs seems to be quite the subset of that I worry about how well this scales and I don't know enough about search libs to even evaluate that.
Is there a good place to start learning about what kinds of situations Typesense works best in (besides needing a Levenshtein-based search), versus what kinds of situations it wouldn't work well in (and perhaps what other libraries would work better)?
Other libraries like Elastic offer more customization but also has a steeper learning curve.
Talking about fastest time to market this is the biggest one rather than setting up elastic, which annoying as it is is still faster than creating the UI.
Does it do normalization as part of the typo search (in case of missed/incorrent accent marks, etc)?
Does it do stemming at all? For English or other languages? (ie, I search for "run" and you show me documents for "running" or the other way around).
Any support for Chinese text (which typically doesn't have whitespace between words)?
While it does not support stemming, with fuzzy prefix matching, it will largely work and practically more useful.
No typo or fuzzy correction for Chinese text yet.
I think mentioning any of them would be okay.
- if using an orm have a hook in your orm model to update the search database whenever a database entry updated/created/deleted.
- if not using an orm, update your rest api/view/any code that does CRUD to update the search index after successful data update
- create a command line tool that sync all existing data to the search index. Probably only used a couple times when initializing the search index with existing data, but it's pretty handy for testing purpose.