Hacker News Reader

Comment by thedarkproject | Hacker News Reader

Would recommend looking into ElasticSearch too. Recently had my first hands-on experiences and I am pretty amazed.

Not sure about 2GB limit though. It doesn't work well at low RAM. Probably better to just use your own suffix tree or some trie for all the stuff you want to search for and use that. I bet you can get it down to something like 100MB.

Depending on the index size (and type of queries, and schema, and expected performance...), 2GB might be more than enough. Solr does a very good work in caching results, query filters and so on in memory. Apart from that, it's disk based.

Of course, performance won't be stellar if it has to go to disk to find every result, but even with that you should be able to get sub-second results with some tweaking.

He didn't really define what he was going to "search" for, so yeah, when meaning full text search, Elastic is probably the way to go. When going for exact lookups, Aerospike will actually make use of his SSDs, when going for slice & dice on wide rows, BigQuery or Redshift will fit his bill - unfortunately he didn't say much about his exact use case. But inventing your own data store is probably one of the worst ideas ever except for gaining knowledge / as a rite of passage.

ElasticSearch (and Solr and Lucene) can be used to do index (and lookup) exact expressions, e.g. string keys/enums, or numbers. It can work with the combination of text search and exact search terms. I've had great results of using these for general purpose search and database lookups.

Reply on news.ycombinator.com