Current architectural best practices for LLM applications | Hacker News Reader

Current architectural best practices for LLM applications

mattboegner.com

141 pointsburakemir24 comments

This is super helpful. I'm building a document question-answering service over a custom data corpus (related to Saivism, a sect of Hinduism). So far the first pass has been to manually chunk the text (based on headings, chapters etc.) and then I've used OpenAI's embedding service and storing the embeddings in Pinecone. All stiched together using LangChain. To ask a question, the question is again embedded, then searched against the vector store, then the related documents are provided as context to the LLM along with the question.

So far it was really easy to set up the prototype, but the results weren't as great as I had hoped, so I'm excited to see how I could improve it.

Edit: wow, I didn't see this before. LangChain implements one of the featured article's suggestions (HyDE) - https://python.langchain.com/en/latest/modules/chains/index_...

This is the same approach that Supabase Clippy took: https://supabase.com/blog/chatgpt-supabase-docs

They called it "context injection" but the OpenAI community appears to call it "retrieval-augmented generation".

Edit: I see now that Matt's post is talking all about these ideas. I'm happy that the terminology around this space is solidifying.

(Tangent) I will go to the grave continuing to call it Supabase Clippy even though presumably this prediction from the Supabase blog post became true:

> Today, we're doing our part to support the momentum by releasing “Supabase Clippy” for our docs (and we don't expect this name to last long before the lawyers catch on).

The lawyers haven’t reached out yet! Too distracted with all the LLM-generated emails they are getting sent no doubt

Why call it Clippy at all if you anticipate a conflict? It feels immature.

yes, it's immature, however in January not many companies had created something like this and it was the most analogous way of explaining the functionality. it's also memorable

depending on what questions you're asking, you could check out LlamaIndex query capabilities - can define different index structures for different queries + can plugin to your langchain workflow: https://gpt-index.readthedocs.io/en/latest/use_cases/queries...

This is one of the areas of LLMs that I find most interesting. So far, I've found simple question-answering over vectorstores to be a lackluster experience. In particular, the more information you embed and stick into the vectorstore, the less useful the system becomes as you are less likely to get the information you're looking for (especially if the users don't understand their queries need to look like the docs the want to ask about.

I haven't had a chance to try out hypothetical embedded docs yet, but I expect they only provide a marginal improvement (especially if QAing over proprietary data or information).

I'd love to see any other interesting, more up-to-date resources anyone has found on this topic. I found this recent paper interesting: https://arxiv.org/abs/2304.11062

> In particular, the more information you embed and stick into the vectorstore, the less useful the system becomes as you are less likely to get the information you're looking for

Can you explain that? I don't follow why it would become less useful

It becomes a ranking problem in a sense. Lots of data that can be the answer and lots of context that “could” be relevant to put into the context window but then you have to pick the right context and answer with the most correct information which becomes less clear as your dataset increases.

What vector store did you use? Was it an issue with the vector store or just the algorithm ANN is just not good with large datasets?

This is it. One of the "apps" I built was a slackbot for my classmates (in business school) that allows users to upload docs via slack (course notes, cases, etc.) that get embedded and you can then QA over in slack. I also added lots of hard-to-find or disparate information from our school like course reviews, registration information, calendars, etc. so we could all access it from one place.

The problem is once there are 10, 20, 30 different-but-similar documents in the vectorstore (like business school case studies), then asking the bot "what are the key takeaways from the airbnb case" grabs a bunch of useless embedded documents to provide as context. Yes, I can tell users how to ask better questions but it's a bad user experience and nobody stick with it or tries to understand why their queries don't work.

I could use hypothetical document embeddings but the problem is a lot of the cases or course notes are proprietary or not publicly available, so I would guess that the hypothetical answers the LLM would come up with won't provide much better context.

This was built with langchain + pinecone.

edit: I think smarter people than I are working on a lot of better ways to do this, but I think one potential solution is to apply metadata to each document when embedding it (e.g., ask the LLM to apply any number of X preset metadata tags) and then, when retrieving context from the vectorstore, filtering the results by those tags.

Thanks for sharing the detail, that's really helpful and I realize I've been facing a similar issue!

> when retrieving context from the vectorstore, filtering the results by those tags

How would you determine what tags to filter by? Would you also need to rely on the LLM to say "which tags from the collection match this question"?

> The problem is once there are 10, 20, 30 different-but-similar documents in the vectorstore

Sounds like a de-duping problem. Maybe use vector embeddings to find near identical documents and limit them in the context. i.e. maximize the vector distance between your context sources.

I wonder if that's the point when fine-tuning becomes a more appropriate option?

This document seems to have been written before the toolformer paper[0], which fine tunes the model to use tools (e.g search) to retrieve information.

[0]: https://arxiv.org/abs/2302.04761

I think that's something different.

Toolformer looks more similar to ChatGPT Plugins (it wouldn't surprise me if ChatGPT Plugins was partly inspired by that paper in fact). It's a way of teaching a model to call tools when it needs to - see also the ReAct pattern.

When you're implementing Q&A on top of a LLM that's not necessarily the right approach. You don't need the model to be able to make searches itself - you need a good strategy for searching for the most appropriate content that fits in the prompt, then sending that to the prompt along with the user's question.

They're both really useful techniques, but I don't see toolformer as a replacement for retrieval augmented generation.

Few other helpful options recently added to Langchain:

1. Extraction for query filters - https://twitter.com/hwchase17/status/1651617956881924096?s=4...

2. Contextual compression to eek more out of prompt stuffing - https://twitter.com/hwchase17/status/1649428295467905025?s=4...

And then it’s been there’s existing great utility chains for map-reduce, with re-ranking, etc for more ways to apply LLM completions over large documents and/or large sets of documents: 3. https://m.youtube.com/watch?v=f9_BWhCI4Zo

We are going to need better retrieval methods as LLMs augment and generate more content on the internet.

I hear this a lot but I don’t know I fully buy it. The LLM have no agency of their own, so someone has to instruct it what to write about. For instance say you’re writing a new article about a Trump trial. You can’t simply ask it for an article about the trial. It doesn’t “know” as it’s new knowledge never written about before. You have to tell it the situation and the specifics gained from first hand experience. That knowledge is encoded in the LLM output. Future LLM being fed that textual information gain the knowledge the human entered and the LLM encoded.

The value the LLM provides in this scenario is simply the language and grammar around knowledge a human provided. It didn’t generate any content whatsoever. The human did that. Similarly a human with a calculator generates numbers but the intent and meaning is provided by the human. The future “LLM all talking to each other” I think doesn’t actually exist, until perhaps multimodal AIs can describe the environment around them. But even then, it’s not an echo chamber because they’re encoding the unfolding of events and time that exists outside the LLM in a way future LLM can encode it.

LLMs can’t create true statements about events not described in their training data, but that’s no reason to assume that they won’t be increasingly prominent on the internet. Advertisement views pay the same whether they are next to true information or merely true-sounding information.

That’s not actually true, high quality news sites make better returns on views and click throughs hence can charge more for advertising. What makes them high quality is both the quality of writing, which is commoditized by LLM, but also the quality of the fact finding and truthfulness. LLM creators don’t have to train their models with known cesspools of SEO garbage pumped out to dupe search engine users whether by humans or machine. Just because a news paper is entirely written by an editorially controlled LLM fed story outlines by reporters doesn’t make the news any less quality than one artisanally crafted by human writers - the facts are what matter, not that the prose was constructed by a writer. In that case, the news as written by the LLM encode facts provided by a human, and would make high quality material for another LLM. I would anticipate there will be LLM who specialize in high quality source material, hence I don’t think an internet of material written by LLM with human guidance would produce any worse LLM than the ones made today.

Sentence embeddings have been great for improving semantic search, but I am still struggling with finding relevant documents for numerical values. Questions like "what people where born in 1992" or "people with at least 4 children". One thing I can do is pre-process the data by transforming the date of birth into boomers/zoomers/millenials and the like but this does not help on the question side if people don't know what to ask

Is this architecture what autogpt uses?

Reply on news.ycombinator.com