I’ve done something similar, but using duckDB as the backend/vector store. You can use embeddings from wherever. My demo uses OpenAI.

https://github.com/patricktrainer/duckdb-embedding-search