I need full-text search for a large amount of text data (500gb++).
I've looked at Algolia, Meilisearch, Typesense, Elastic. From what I can tell, they all require a server which can keep everything in memory.
MySQL full text search is probably sufficient, so maybe I'm asking about best practices to store a MySQL database at terabyte scale.
Thoughts on how to scale a database and do faceted full-text search without it costing a ton of $$?
Key points:
- Columnar Storage: Efficiently handles large datasets on disk, ideal for terabyte-scale data. It's not enabled by default but can be set up easily with "CREATE TABLE ... ENGINE='columnar'".
- Faceted Search: Probably easier than anywhere else with just "FACET <field name>" added to your "SELECT" query.
- MySQL Protocol and SQL Support: If you’re familiar with SQL and MySQL, it's easier to get started compared to other search engines.
That way I can get distributed indexes on S3 for cheap. Thanks for pointing the way!