Hacker News Reader

Comment by weinzierl | Hacker News Reader

How would I decide on which granularity level I create my vectors? For example, if I create word level embeddings with word2vec and store one vector per word I probably will only get good results for keyword based queries, right?

If I create more fancy vectors, for example on the sentence level, it is not really clear to me how that is supposed to work.

Why is the vector embedding of a query, which typically has the sentence structure of a question, near the vector embedding of a propositional sentence that answers that query? Sure, it will probably not be completely off, just for the fact that query and answer will contain similar words, but how can that be better than just a fuzzy word search?

And finally: Should I decide on on particular level (like sentences) and store that or should I store word2vec and sentence and paragraph vectors in the same collection?

View on HN

Check out the chapter on embeddings in this wonderful book: https://web.stanford.edu/~jurafsky/slp3/

I would not use word2vec in any application today - I used it in the blog post because it's a well-known model and because the embeddings it generates are static, i.e. one token always maps to the same embedding regardless of context.

If you want to create embeddings at the sentence level, a good place to start is SBERT: https://www.sbert.net/

I understood that it was just an example and it is a good choice for an introductory text. My question is more into advice how to go on from that. For example, is SBERT a good choice if most of my queries are in fact multi-sentence paragraphs?

I guess what still doesn't add up in my mental model is how people seem to assume that the query embedding vector must somehow automatically be near good answer embedding vectors. I would have assumed that some conditions have to be met to make this true and I would be interested into advice in that regard, or alternatively an explanation why I'm wrong with my assumption.

You’re correct. There are lots of ways to project a string of embeddings back onto a single embedding - lots of times however these simpler methods do something like pooling the sum if the word embeddings by averaging and hoping that the answer embedding has a small angle with that projection (cosine similarity). You could try adding a way to focus on individual tokens, pairwise interactions, and more sophisticated ways of pooling and projecting embeddings, and then boom you’ve basically just created transformers.

oh boy, this looks like an alchemy to me. on one hand one cant deny success of llm's on another we are shifting the responsibiliteis to non-deterministic fuzzy duck-taped functions.

My impression is similar and that's why I asked the question.

Another thing that is not clear to me: Is there query directly fed into e.g. SBERT or should I ask an LLM to transform the query into something more suitable, like turning the question into a proposition?

Asked more abstractly: In a vector space like SBERT's, can I expect questions and answers about the same topic to lie near each other? Especially will the correct answers lie near their question?

>Another thing that is not clear to me: Is there query directly fed into e.g. SBERT or should I ask an LLM to transform the query into something more suitable, like turning the question into a proposition?

This not how 99% of embedding models work(though you can train for specific tasks) but as it turns out such a thing is possible and is beneficial.

Instruct tuned embedding model here - https://instructor-embedding.github.io/

This kind of thing is going to seem so obvious in hindsight in six months once all the sota methods start converge upon similar improvements lol. It’s still so early, I keep telling myself.

> I would not use word2vec in any application today

Sbert needs to be finetuned to get anwhere near good results if the training objective deviates from the original one. Word2Vec can be estimated from collections of tokens in an unsupervised fashion, so it definitely has its place even today.

Reply on news.ycombinator.com