Hacker News Reader

Comment by Palmik | Hacker News Reader

It seems unlikely that raw llama2 will perform better than purpose made encoder models like bge [1], gte [2], e5 [3] or instructor, despite it's much much larger size (for the tasks people usually need embeddings for).

You can probably get it to behave well with a fine tuning like this: https://arxiv.org/pdf/2202.08904.pdf

[1] https://huggingface.co/BAAI/bge-large-en

[2] https://huggingface.co/thenlper/gte-large

[3] https://huggingface.co/intfloat/e5-large-v2

View on HN

This is my sense as well. Text generation LLMs haven't been the best source of embeddings for other downstream use cases. If you're optimizing for token embeddings (e.g., for NER, span detection, or token classification tasks), then a token training objective is important. If you need text-level embeddings (e.g., for semantic search or text classification), then that training objective is required (e.g., what Sentence BERT did to optimize BERT embeddings for semantic search).

That's a great list of existing embeddings models (in addition the SentenceBERT models https://www.sbert.net/docs/pretrained_models.html).

The SGPT model is a very high performing text embeddings model adapted from a decoder. Using the same techniques with Llama-2 might perform better than you expect. I think someone will need to try these things before we know for certain. I believe there is still room for significant improvement with embedding models.

Thanks for pointing out those models. I see from a quick Huggingface search that the bge model is available in GGML format. You can trivially add new GGML format models to the code by simply adding the direct download link to this line:

https://github.com/Dicklesworthstone/llama_embeddings_fastap...

So to add the base bge model, you could just add this URL to the list:

https://huggingface.co/maikaarda/bge-base-en-ggml/resolve/ma...

I will add that as an additional default.

I think it's still overkill though for semantic embedding, SBERT is on order of ~250mm parameters while smallest llama at 7b parameters.

If all you want to do is make some basic semantic search, that’s probably true. But I strongly suspect we are only just now starting to scratch the surface of what’s possible with embeddings that come from much more powerful LLMs like Llama2 that can clearly manifest much greater demonstrated “understanding” of sentences they are shown (whatever that means, but intuitively, it seems obvious to me). That’s partly why I made this tool—- to aid in my investigations of LLM embeddings in a convenient and performant way.

I'm really curious to see where that investigation leads - have you done any comparisons between Llama 2 and the embedding focused models? I wonder if it'll be better able to provide more 'intuitively correct' similarities?

Yeah, I'd be surprised if embeddings derived from decoder-only models are competitive in common embedding tasks without some extra training work. There's a good benchmark page on huggingface for the MTEB tasks (Massive Text Embedding Benchmark) that's kept up to date here: https://huggingface.co/spaces/mteb/leaderboard

I did not know about BGE, learned about it from your comment. Seems to be the new SoTA for semantic embeddings, even for small models. Very cool!

It's exciting/terrifying to see how fast all this moves, it feels like every day there are new discoveries, techniques and models.

When I talk with people about ChatGPT-esque things (ChatGPT is pretty much what most people know now), I say that it's crazy that you can run this on your own consumer-level hardware if you just want to mess with it. You don't need "prosumer"/enthusiast hardware (unless you want to train models, but then I see people are using Google Colab etc).

It's a crazy world we live in.

Reply on news.ycombinator.com