https://docs.meilisearch.com/learn/advanced/storage.html#lmd...
>For the best performance, it is recommended to provide the same amount of RAM as the size the database takes on disk, so all the data structures can fit in memory.
> [...]
>It is important to note that there is no reliable way to predict the final size of a database. This is true for just about any search engine on the market—we're just the only ones saying it out loud.
Looks like a 10MB document is taking ~200MB, from their docs. I don't think that scales linearly though, since it's a reverse index it is going to scale based on the number of unique words it finds, with each document adding a bit on top of that. You'd expect it to have a pretty big index to cover common english words, and then each document adds a bit on top of that.
Definitely seems like somewhere they could make some improvements though. Some transparent compression could probably help, and with zstd's dictionary feature it can be fine tuned to the data they're actually seeing.
Not about to replace xapian in kiwix (offline wikipedia reader) any time soon, I think.
With manticore, we've tried to run into these issues in benchmarks, but the only problem we got was temporary high IO load when indexes need to be re-indexed with new or changed documents. In total it's at 50-70% of the RAM usage compared to Meili.
We'd be happy to re-visit, but looking at the docs - it seems to be about the same as it was back then (a year ago).
And indeed, Meilisearch uses memory-mapping, which means that everything is on disk, and it will try to take as much memory as possible. For your information, we successfully ran a 115M documents dataset on a 1Gb RAM machine.
[1] https://manual.manticoresearch.com/Introduction#Storage-opti...
Would be nice if you could check a query and then start the instance with an appropriate memory configuration.