Have you considered using a proper vector database, so you can scale it up and run in a cluster? I mean something like Qdrant.