Thank you so much, we will definitly add these comparisons!
318 karma · joined January 27, 2019
Sonic is "just" a inverted index with a levenshtein automaton, it will returns only the documents ids in which there is your requests words and then you will have to retrieve the full documents in your main database and only then can you apply some relevancy ranking by yourself.
MeiliSearch does not offer faceted search yet. It is one of the key feature that we are still missing but we plan to work on it in the coming weeks.
For your use case today, I suggest you use Typesense if it fits your needs ( they handle faceted search already ) or Tantivy or Toshi.
To manage different languages, you should make one index per language. MeiliSearch and Tantivy handles kanjis! We will add faceted search in the coming weeks if you are not in a hurry :D
based on http://highscalability.com/blog/2013/1/28/duckduckgo-archite...:
The fat tail queries go against PostgreSQL and the long tail queries go against Solr. For shorter queries PostgreSQL takes precedence. Long tail fills in Instance Answers where nothing else catches.
It seems that Bing is now a part of their sources indeed: https://help.duckduckgo.com/results/sources
Lucene and Lucene-like projects (Tantivy or Bleve in Golang) are general-purpose search libraries. They can handle enormous datasets, and you can make very complex queries on them (compute the average age of people named Karl in a certain type of document for example).
These libraries are based on tf-idf (term frequency inverse document frequency) algorithm and manage quite poorly typos for example (unless you make the setup to index your documents differently to parse them correctly).
Toshi is like Elastic for Lucene, it provides sharding and JSON over HTTP api.
You can basically used Lucene and its derivatives for basically any search related project, but you may have to dive into how it works and understand concepts like tokenization or ngrams to tune it according to your needs.
On the other hand, MeiliSearch (and I guess Typesense, but I can not talk for them) focus a subset of what you could build with Lucene or Elastic.
It is a fully functionnal Restful API, made for instant search or search-as-you-type. The algorithms behind MeiliSearch are simply different: a inverse index, with a levensthein automaton to handle typos, then a bucket sort you can tune for the ranking of the returned documents. The aim is to provide a easyer go-to solution to implement for customer-facing search.
You won't be able to make super complex queries on terabytes of data. We just make super fast and ultra relevant search for end-user.
TypeSense and MeiliSearch focus on the same usage, we choose Rust for performance, security and the modern ecosystem that will allow easier maintenance :D
Both MeiliSearch and Typesense use a reverse index with a Levenshtein automaton to handle typos, but when it comes to sorting document:
- Typesense use a default_sorting_field on each document, it means that before indexing your documents you need to compute a relevancy score for typesense to be able to sort them based on your needs (https://typesense.org/docs/0.11.1/guide/#ranking-relevance)
- On the other hand MeiliSearch, uses a bucket sort which means that there is a default relevancy algorithm based on the proximity of words in the documents, the fields in which the words are found and the number of typos (https://docs.meilisearch.com/guides/advanced_guides/ranking....). And you can still add you own custom rules if you want to alter the default search behavior.
You can enable your full network access to the API by launching it like this `meilisearch --http-addr=0.0.0.0:7700`. The default listening address is 127.0.0.1 which is why you can only touch it from localhost. You can also pass it as env variable, you can read more on the documentation https://docs.meilisearch.com/advanced_guides/binary.html#env...
And yes, MeiliSearch does not ingest PDF or DOC out-of-the-box, it's a RESTful API to which you can send JSON. You might want to look at https://github.com/dadoonet/fscrawler, but it is working only with Elastic atm.