Hacker News Reader

Comment by simonw | Hacker News Reader

The trick I describe in this article - using embedding similarity to find relevant content, then constructing a prompt to try to answer a question - is showing up in a whole bunch of different places right now.

https://github.com/jerryjliu/gpt_index is a particularly interesting implementation under very active development at the moment.

View on HN

There was a 2020 paper called Retrieval Augmented Generation that describes this technique too [1]. That paper used BERT embeddings which are a little cheaper and uses BART as a generator.

One of the common failure modes of RAG (and I assume your technique as well) is hallucination: basically, making stuff up that isn't in any of the docs [2].

[1] https://arxiv.org/abs/2005.11401

[2] https://parl.ai/projects/hallucination/

I was playing around doing the same thing today but using Pinecone [1] as a vector store. Some other folks were experimenting with building key-value stores at the Scale AI hackathon this weekend [2]

[1] https://www.pinecone.io/

[2] https://twitter.com/sjwhitmore/status/1617318051455840258

One tip to speed up how you're finding the closest matches -- openai's embedding vectors are normalized and all fall upon a sphere. So, the cosine similarity is equal to the dot product. Might speed things up a bit to use only the dot product.

The attention mechanism in transformer models also uses this trick - just dot product. I learned this during a long conversation with ChatGPT the other night - an excerpt of its reply to me:

  The dot product between the query and key representations is similar to computing the cosine similarity between two vectors. The cosine similarity is a measure of the similarity between two vectors in a multi-dimensional space, and is defined as the dot product of the vectors normalized by their magnitudes.

  The dot product of the query and key representations can be seen as an un-normalized version of the cosine similarity, in the sense that it computes the dot product of the two vectors. The result is a scalar value, which represents the similarity between the two vectors, the larger the scalar, the more similar the vectors are.

This approach also has limitations. Namely, your ability to retrieve information is limited by your ability to search for snippets using embeddings.

This could be solved with using different search methodologies, using multiple gpt requests to summarize available info or using a structured knowledge framework to prepare prompts (instead of just raw text).

Any other ideas that I'm missing?

There's so much scope for creativity and improvement here - that's one of the things that excites me about this technique, it's full of opportunities for exploring new ways of using language models.

gpt_index does that. A tree of document chunks (leafs) is built with parent nodes as increasingly summarized versions of the child nodes built with GPT.

The tree is then traversed to find the most relevant chunk asking GPT to compare entries based on relevance to the question. This results in an original document chunk, which is given as context in a final prompt asking to answer the query.

This is great and powerful, but very not cost effective. Log(n) requests to completion API, for n documents.

The embedding search is probably necessary for bigger datasets.

In my experience semantic search is great for finding implicit relationships (bad guy => villain) but sometimes fails in unpredictable ways for more elementary matches (friends => friend). That's why it can be good to combine semantic search with something like BM25, which is what I use in my blog search [1]. N-gram text frequency algorithms like TF-IDF and BM25 are also lightning fast compared to semantic search.

[1] https://lukesalamone.github.io/posts/rolling-my-own-blog-sea...

It's a great write-up Simon

Reply on news.ycombinator.com