Search engine needs have evolved a lot since Lucene was designed (Lucene is the search library behind ElasticSearch and Solr). Users now want an instant search that search at the first keystroke and are able to tolerate typos.
Lucene does not propose anything special for that, so you have mainly two options:
- slow down indexing: index all ngrams of words (this also require a lot of disk)
- slow down search: you need to find all words that match the prefix query in the dictionary and then retrieve all inverted lists.
ES and Solr have added autocomplete module that is basically a compiled Trie that associate a string to a results set, but this is not a real instant-search engine, there is no intersection of inverted lists and this produce mismatch between the autocomplete and search results.
On our side, we have redesigned all search algorithms and data-structures from scratch to run on a mobile devices. We are for example able to search all cities from Geonames (> 3M) on a old iPhone 3GS in less than 50ms (including prefix search & typo tolerance) with only 1KB or RAM and a flash disk. You can't do that with Lucene. So I let you imagine what we can do with a high-end server !
For the High-availability & Consensus, this is something complex and we have spent a lot of time to test it by killing machines of a cluster, and we still continue to test it all the days since every deployment kill index builder process !