Having supported Solr/ES/Lucene in production for 4+ years now (websolr.com / bonsai.io) I would be pretty hesitant to trust Lucene in general as a primary data store. Beautiful for secondary indexing, but otherwise, Why Not Postgres?™ ;)
EDIT: I don't think MongoDB is there yet either. There are definite benefits and drawbacks between Postgres and ES, tipping heavily towards Postgres for structured heavy write data. But for ES and MongoDB? I think MongoDB falls a bit short there.
But I personally suspect Lucene won't ever get away from the dreaded "just reindex." And to the larger point, I think recent resurgent interest in data stores and distributed systems have shown pretty clearly that there is no holy grail. No single data store can provide all the semantics necessary for all use cases. Maybe not even for most use cases. There are just too many tradeoffs to consider.
Believe me, I earn a living hosting Elasticsearch, so I'd love to see it become a robust primary data store. There are some use cases where it actually does make sense—just look at the amazing traction ES is experiencing for storing and indexing time-series data.
But as a general-purpose primary store, I'm not really holding my breath. Maybe I'm just becoming battle-worn and bitter. I would love to be proven otherwise over the next few years!
ES is suitable for full-text based document indexing for enterprise level or any websites, which means they have a reasonable amount of data to be indexed in a given timeframe. A complete re-indexing won't not take for a couple of days.
So the basic idea behind the NoSQL database is to dump the data into the database quickly and return, so you can see very fast response for insert and delete. Then it will load the data into the memory to process for real-time retrieval which also produces fast response from select. I'm not sure about update.
If the data volume grows, they quickly add shards or make the number of pre-shards big enough to allocate enough memory resources to handle the queries or let the OS to swap the memories by adding more server nodes.
So if you want to use NoSQL database, you must be bound with the system requirement and make your application fit into that and take the most advantage from it. Otherwise, if you are running high structured data store, better to use relational database.
Another point is: if the documents are collected from the web like search engine, NoSQL will not fit for the large volume of data and relational database is also used to store the indexed data for fast retrieval. I guess this is what you meant "general-purpose primary store".
Correct me if I'm wrong.
For example, Postgres lets you reason about integrity, atomicity and transactional boundaries, and whether things are really safely stored with synchronous replication. If Postgres returns after a commit, I trust it. However, that requires me to have two servers working, which is harder to keep highly available.
ZooKeeper, on the other hand, I can rely on being available. But that's not really something you want to be putting lots of load on, nor try to do anything but trivial "queries". And the more servers you add, the slower writes get.
I don't trust Elasticsearch enough for those tasks, yet I wouldn't want to do searches in Postgres (Yep, I'm familiar with tsearch) even though it can. Elasticsearch is simple to scale out and awesome for searching.
Logs and metrics we shove straight into Elasticsearch, however. Other things go from ZooKeeper to Postgres and then to Elasticsearch, or from just Postgres to Elasticsearch.
Separate tools for separate jobs. I'm one of the co-founders of www.found.no, one of the hosted Elasticsearch providers . We absolutely love Elasticsearch and find new use cases for it all the time, but it's not going to be the one store to rule them all, at least not very soon.
Indeed!
That said, it's great that more people are picking up Elasticsearch for new exciting things.
Elasticsearch has really pushed what constitutes a "search problem", and deserves lots of kudos for that! :)