Are there any good document embedding models I can run on my own hardware? Word2Vec is interesting, but I'd like to also try out cross-linking blog entries, etc.
pip install InstructorEmbedding
pip install torch
pip install numpy
pip install tqdm
pip install sentence_transformers
from InstructorEmbedding import INSTRUCTOR
import numpy as np
from scipy.spatial.distance import cosine
xl = INSTRUCTOR('hkunlp/instructor-xl')
embeddings = xl.encode(["I'm amazed at how autoencoding actually works, despite the simplicity of the approach.", "Are there any good document embedding models I can run on my own hardware? "]).tolist()
cosine_distance = 1 - np.dot(embeddings[0], embeddings[1]) / (np.linalg.norm(embeddings[0]) * np.linalg.norm(embeddings[1]))
print(cosine_distance)
0.33516267980261427More about the tooling I use to run embedding models locally here: https://simonwillison.net/2023/Sep/4/llm-embeddings/