Kerollmops

Thank you for the feedback. You should probably look at our blog post explaining how to index in smarter way [1]. You should probably come and talk to us on Discord. We work with MrBeast and they don't have any issue indexing millions of YouTube videos everyday.

Ho! And just to tease you a little bit. We highly improved the indexing speed of Meilisearch v1.6 (january 24). Here is a teasing tweet [2].

[1]: https://blog.meilisearch.com/best-practices-for-faster-index... [2]: https://x.com/kerollmops/status/1734576622303404317

Indeed, you are right about this. This is why we will release geo-replication on the cloud in Q1 next year. We heard you and worked hard on both of those features. We already have a great working proof-of-concept.

It's my own company and I work hard on both Meilisearch, the keyword search part since 2018 and the soon-to-be-released semantic search part. The hybrid search will be also part of the v1.6 release in january. I took time to write those three blog posts.

What do you think about the blog posts content?

That looks pretty cool. Would you mind explaining a little bit how you did that into more detail? Like, you ask Meilisearch some documents (keyword search) and then ask the LLMs score of those and refine the list a bit?

> Yup, that's exactly what I'm saying there is something wrong with data model.

We are an Open Source project and know we can do better on the indexing speed. We already did much work on that subject. We enabled the auto-batching feature, which significantly improved the time to index documents.

Are you sending all your documents in one go, or are you waiting for the tasks to be indexed to send the next batch?

> I also noticed it's only using a single thread.

Unfortunately, one pass of the indexing process is currently single-threaded. We can do better, and we are aware of that. Software development takes time.

> Out of curiosity what dataset size you guys are testing on?

We are testing on a broad list of datasets, going from 200k movie datasets to 142 million song datasets.

> Other learning is, not every software written in Rust is fast :-)

It depends on what you want to be fast. Software programming is a matter of trade-offs, and we have excellent search speed results.

Hey rkwasny,

Which version of Meilisearch were you using?

We chose to use LMDB instead of RocksDB because it is a much faster, memory and CPU-efficient key-value store. However, it would probably be much quicker to insert all those inverted indexes into RocksDB. RocksDB is an LSM tree, meaning that merges must be done to retrieve the different entries, which means that the search side of the engine would suffer from that. LMDB is much easier to operate. There are a lot fewer parameters to manage the user-end cache, etc.

The bottleneck of the Meilisearch indexing phase is not writing into LMDB. It is mainly related to computing a bunch of databases storing prefix words and the computation of other databases. We are working on making that much faster and reducing the index size on the disk.

Co-founder and Tech Lead here,

Thanks to Louis' work and the design of LMDB, we can efficiently use the virtual address space of the OS to let it manage in the best way the memory Meilisearch is using.

We continuously improve the indexing speed, index size, memory usage, language support, and search speed. Perfection is a goal, and we will eventually reach it. Unfortunately, computer science is the science of trade-offs, and that's the hard part: choosing the most critical feature. We decided that search speed and accuracy are.

We are not in contact with beir or the owner of the bei-cellar oganisation.

However, we started tracking our relevancy with the TREC 4 & TREC 5 data which are provided by the NIST organisation [1]. I can only tell that the results are very good and that we continue to improve that. We will talk about that in a blog post.

[1]: https://nist.gov

What’s funny is that (1) doesn’t look like a real limit when you know that the first Harry Potter book is nearly 77000 words. The recommended way is to split your documents by paragraph to increase relevancy, this way you can see the exact part that match.

About (2) we will work on exposing two new ranking rules to be able to control that.

For (3) I thought it was fixed.

We decided to implement (4) the PUT and POST this way after looking how others were doing that.

Thank you very much for this amazing feedback, really appreciated.

We did a lot of improvement to the indexing part of the engine and now can auto-batch updates which gaves incredible improvements. We will continue to work on this in 2023. Can I know the version you were using?

I am the co-founder and maintainer of the engine, and I confirm we have some localized unsafe blocks for when we interface with the C library: LMDB.

However, I prefer having a few unsafe blocks that I can review carefully than a single one encapsulating the primary function.

A previous version of MeiliSearch was using RocksDB and we were having a lot of trouble using it, a lot of setup to do to make sure that we were not killed by the OS due to OOM or even fixing a lot of strange segfault by patching the RocksDB library itself...

RocksDB doesn't support transaction but views in the database, which means that if you are indexing, writing into the database and that any event makes your program to stop unexpectedly, you can't just start your program and use the data like this as it could be corrupted.

This is why at MeiliSearch we prefer using LMDB, even in case of an unexpected crash, a reboot is instant and valid, you just need to restart the indexing you were previously doing and can serve requests to the users with the previous version of the database.

Also as you can see [0], the benchmarks between LMDB and RocksDB is very clear. I understand that it is maybe not reading the database that takes time on your side but it is on our side, combined with the set operations between sets of internal documents ids.

[0]: http://www.lmdb.tech/bench/optanessd/imdt.html

As we don’t use RocksDB but LMDB, we use a lot less real memory than key-value stores that uses a user-side cache system. LMDB is memory mapped and therefore let the OS manage memory for it. Typesense uses RocksDB and ElasticSearch a custom key-value store, used by Lucene internally.

The real advantage of LMDB is that it is a BTree, key-values are ordered and do not need any computing when retrieved which is not the case of a LSM-Tree key-value store like RocksDB that needs to merge/compact pages of key-values pairs before being able to return it too you. Wasting CPU when the search engine must use its CPU to do union/intersection…

Another advantage of LMDB is that it returns a view into the DB itself of the entries, RocksDB can’t as it must do operations on the entries before returning them to the library user, for example: decompressing or compacting the values.

Hey, this no more a prototype, it is the internal engine under MeiliSearch. I forgot to update the README :)

Nice work, it’s pretty awesome to be able to plug a search engine as easily as adding a plugin. I just have one question about the recently published benchmarks, are the latency based on the two-word search "hello world"?

I would say that you must add your own nginx (or else) in front of our HTTP only engine, in term of fault tolerance we are working on high availability.

To make MeiliSearch expose the documents that are stored in your PostgreSQL (or any other database) you must extract them and store them in our engine using the HTTP API we provide to you. https://docs.meilisearch.com/references/documents.html#add-o...

For that you will need to also define the different attributes your document is composed of.

We thought about providing a simple tool to extract the documents from an SQL table into the MeiliSearch directly.

This is not something that MeiliSearch supports currently but I am working on making the engine be able to index other formats than JSON, I saw great performance improvements when indexing simple CSVs.

We will probably make MeiliSearch accept different indexable formats (i.e. CSV, JSON, JSON-lines) in a future version.

Just to add a little note here, we are currently working on the functionality of multi-filter queries, because we are aware of our community!

I understand what you mean but is it for a specific usage of a search engine? I was thinking that this kind of time-restricted tokens could also be managed by the nginx instance, our engine doesn't support that for the time being.

Go on and try MeiliSearch, 50k documents are easily handled by the engine and with not much RAM usage.

It will take you something like 10 minutes to start and populate MeiliSearch, you will be able to test it just by going to the server HTTP url in no time!

Currently we think this kind of security can be enabled by a simple nginx setup, allowing autorefesh of certificates easily (e.g. certbot). But in the future we will probably handle that in the engine itself.

Yes, there is, you can find all clients on this documentation page: https://docs.meilisearch.com/resources/sdks.html

Note that we are reworking the js library and there will probably be React integration too!

MeiliSearch is "zero-config" compared to ElasticSearch in terms of setup to make it work for end-user instant and relevant search engine. Our engine follows the Algolia engine in terms of typo-tolerance, relevancy, and speed.

Here is a little comparison to enlighten your questions: https://docs.meilisearch.com/resources/comparison_to_alterna....

The current API key system is a simple and temporary solution.

We will work on a more feature-full API key system including the one you are talking about. This is on our roadmap IIRC.

It remind me of the FTL OST Good for programming too !