How many dimensions can you use this with efficiently? Could you use this to store embeddings from machine learning models and find nearest neighbours of various items, like aNNoy or faiss does?
It's a pretty cool extension that does distance between points, intersections between n-dimensional cubes (hence the name), different distance metrics etc.
It'd be perfect for storing and searching through large amounts of n-dimensional embeddings, I'm guessing it's used for that already.
Still wouldn’t recommend it though.
1024 dimensions is a lot! Could you elaborate on what application requires that many? If it's a DNN layer output, your data must be sparse, so dimensionality reduction won't affect your recall if tuned properly.
Thanks a lot for your reply!
The article discusses a low-dimensional KNN problem. The curse of dimensionality guides intuition that the methods here likely will not apply to extremely high-dimensional problems.
faiss actually comes with a lot of excellent documentation that describes the problems unique to KNN on embedding vectors. In particular, for extremely large datasets, most of the tractable methods are approximations that make use of clustering, quantization, and centriod-difference tricks to make computation efficient.
See https://github.com/facebookresearch/faiss/wiki/Faiss-indexes and related links for more information.
https://postgis.net/docs/manual-3.0/ST_MakePoint.html
Edit: The distance operations are 2D by default and it looks like there are only 3D variants.
You can use a python library inside PostgreSQL using plPython, but supplying the coordinates to the evaluation is not going to be as compact and specialized as this