Hacker News Reader

Comment by gk1 | Hacker News Reader

Pinecone does this, FYI: https://www.pinecone.io/docs/metadata-filtering/

And you're not alone... Lots of people want this, and lots of people can't get it with ES.

Yes but my sense it is coming pretty soon for ES, and switching to Pinecone has a huge rewrite cost. As NNs keep getting better, vector search keeps becoming more important, and it becomes more and more existential for elasticsearch to do this well. I wonder if there actually some fundamental reason they won't be able to do so.

We've had people kicking down our doors (figuratively) for the filtering feature. I have no doubt ES is getting the same feedback and is working on it.

> I wonder if there actually some fundamental reason they won't be able to do so.

This is where it gets interesting. I don't think there's anything a company with ES's resources can't do if they really want to. However...

ES's vector search is based on Lucene 9.0, which in turn uses the HNSW vector index. HNSW is shaky when it comes to CRUD updates (namely deletions), and flat out does not support metadata filtering: https://issues.apache.org/jira/browse/LUCENE-10040

Even with ES resources, it's a fundamentally hard problem. In the best case it's going to take a while to implement.

If you control the HNSW implementation, it can definitely do pre-filtering. Vespa does it, and you can modify open source HNSW libs easily. I added pre-filtering support to an internal fork of HNSWLIB last week, for example…

Reply on news.ycombinator.com