[Disclaimer: I work at Cliqz]

We will have a blog post tomorrow on this very topic, but in short, we use a combination of Keyvi, Granne (both in-house) along with Cassandra and RocksDB.

Though our approach mentioned in this blogpost significantly reduces the storage needed to host the index, we still have an index of around 50 TB of data.

[Disclaimer I work at Cliqz] There is a lot of systems under the hood, depending if it's the main index or the freshness index. But if I have to pick one as database it should be Keyvi (https://github.com/KeyviDev/keyvi).

FYI, there will be a bunch of articles regarding search in the next week.