Hacker News Reader

Comment by Layvier | Hacker News Reader

I'd be curious to hear what people think is state of the art of this kind of problem? I think the Cohere embeddings model v3 is very good, as it handles queries and documents differently to embed them in the same vector space. Otherwise for a specific use case I guess dense retrieval (which is basically a problem specific fine tuned version of this approach) is the best way to go about it?

View on HN

> it handles queries and documents differently to embed them in the same vector space

I assume this might be behind the state of the art, then. As of a year ago OpenAI managed to get an embeddings model that can do both without any special flag or dual model to treat queries and documents differently.

handling queries and documents differently leads to an increase in overall retrieval.

If you're talking about embeddings model metrics, the MTEB Leaderboard is a good resource: https://huggingface.co/spaces/mteb/leaderboard

Yes I had a look at it, I haven't tried e5-mistral-7b-instruct yet but I'll definitely give it a go. Is there such a leaderboard only focused on retrieval by any chance? I haven't found one so far

The BEIR project might be what you're looking for: https://github.com/beir-cellar/beir/wiki/Leaderboard

https://huggingface.co/spaces/mteb/leaderboard

see the retrieval tab

It's interesting how the massive model of e5-mistral has only marginal performance gains over the bge-base and similar ones. It could still be useful for the longer sentence length though.

e5-mistral is essentially a distillation from gpt-4 to a smaller model. You can see here https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a69...

they actually have custom prompts for each dataset being tested.

Question would be, if you haven't seen the task before, what is a good prompt to prepend for your task?

IMO e5-mistral is overfit to MTEB

>I'd be curious to hear what people think is state of the art of this kind of problem?

I would assume Amazon's product suggestion would be SotA for reccomending a book based off of another. It is a recommondation system and while it uses semantic search there are many more ranking signals it uses.

The problem with systems like that is they're going to be optimized for making Amazon the most money, not necessarily what's best for the user (unless the two happen to coincide.)

Money is a proxy for user value and it is in Amazon's best interest for the customer to continually act upon these recommendations. If Amazon fails to deliver user value customers will be less willing to continue paying.

> Money is a proxy for user value and it is in Amazon's best interest for the customer to continually act upon these recommendations. If Amazon fails to deliver user value customers will be less willing to continue paying.

You're talking about the same Amazon that will blindly give me recommendations for other products that fill exactly the same niche as the one I just bought.

"Oh, you just bought a generator? You probably need a second one too, right?"

lol. lmao.

Reply on news.ycombinator.com