We built semantic search for ArXiv

sigmoidprime.com

60 pointsgk13 comments

If you're worried about getting charged by OpenAI on every search query and to add each new document, you could look at some open source tools, Marqo (github.com/marqo-ai/marqo) would work for this, and is entirely free and end-to-end

To anyone coming from the article linking to that Twitter thread of absolute frippery: Welcome!

Secondly, I've been looking for something like this for a while! Thanks for writing this up and posting it, I enjoyed reading this and can't wait to see the iterative development that happens as we better wrap our heads around how to efficiently run semantic search. The side effects of GPT-4's release including that the GPT-3 embedding models being so much cheaper (IIRC) is nothing to shake a stick at! Price performance is maybe not always so bad a thing (especially if it's convenient)! :D <3 :)))) :'D

Hope this sticks around and keeps improving in some form! :) :D :D <3

Someone who would rather self-host could use an embedding from huggingface (minilm is designed for this task) and faiss for the similarity search. I would really like to see a cross-encoder trained for search as that would noticeably improve on bi-encoder approach these guys who use.

Reply on news.ycombinator.com