Hacker News Reader

Comment by Noe2097 | Hacker News Reader

Noe2097·

Hi! Yes! Nice article (articles actually).

I realize you are hinting at it as the subject of a later article, but could you share with us a little bit more about single-stage-filtering :D ? how does it work? How can metadata and vector data "coexist" in a same index?

View on HN

Yes as gk1 said, we're figuring out what can be said! I can't fully answer your questions beyond, having both indexes together helps us speed up the search (thanks to a lack of network/serialization/parsing overhead), and because of the structure we store metadata tags in, we also don't need to check the metadata for every sample (hence the cool speedup you get when filtering) - sorry that we can't say anything more yet!

There are different approaches for turning vectors into terms which you can then simply index with something like Elasticsearch. There are papers by the Walmart and Jet.com teams that describe different approaches, starting with both binarised vectors as well as full float vectors. Not saying that this is what Pinecone is using, so it will be interesting to read their follow-up.

I'm from Pinecone and can tell you we are figuring out now exactly how much to say about this.

We know you want to know, and we also know other vector databases want to know -- because this is a highly requested feature and it's far from trivial to implement in a low-latency way. That's why we have to give the admittedly non-satisfying answer of "stay tuned!"

Neural networks.

Reply on news.ycombinator.com