A few days ago, a colleague asked me why a certain Google cloud instance does not exist. I did not know either, so I searched for this name in the Google audit log, and found when and by whom it was decommissioned.
But it was a full-text search, given the instance name. I probably could do it (in theory) as a field match, if I knew which field it was, and which format it was in (I am talking about project/abc/location/xyz type of junk that precedes the actual instance name).
And yes it was slow (this instance was deleted months ago, and Google tries to search the most recent logs first).
Naturally your experiences will be different from mine!
My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields.
Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.
Your complaint really doesn't make sense, how would you query an unindexed field? Elasticsearch is a _search_ engine, which means it needs to index content that is to be discoverable. What you're saying with unindexed fields is you're completely fine with those not being included in any search or filtering.
It's ok to fail if the indexed criteria are not selective enough. In fact it's usually preferable to a long timeout.
Compared to my experience, you have a foundational difference of understanding with how systems are actually used.
Without knowing for sure I imagine they originally expected the application side to handle this but many of the current solutions don’t do that. And they expose and overload the elastic search query language as the primary search interface with no additional app logic. The elastic search query “is” the search application.
Making some assumptions but might reconcile the different viewpoints on why it does or doesn’t make sense.
And I am only talking about server-side costs here as moving data between server and client has costs both in serialization and transmission. Yes, I can make up regexes that wind up throwing this cost comparison out the window (e.g.: lookbacks), but the fast majority of cases this is true.
I think the main reason that ElasticSeach does not do this is that they would either have to provide grep-like or regex support, and those two would provided different answers than the lexical search system they provide otherwise. That would be a nightmare to try and explain the differences to clients.
Note: in most places I wind up using ElasticSearch I absolutely hate that it is lexical search rather than grep or regex... especially when I am looking for exact text. This is particularly a problem in Jira where I have to be very careful about word boundaries.