Txtai: Open-source vector search and RAG for minimalists

Based on what you're looking to do, it sounds like Retrieval Augmented Generation (RAG) should help. This article has an example on how to do that with txtai: https://neuml.hashnode.dev/build-rag-pipelines-with-txtai

RAG sounds sophisticated but it's actually quite simple. For each question, a database (vector database, keyword, relational etc) is first searched. The top n results are then inserted into a prompt and that is what is run with the LLM.

Before fine-tuning, I'd try that out first. I'm planning to have another example notebook out soon building on this.

Fine-tuning is almost certainly the wrong way to go about this. It's not a good way of adding small amounts of new knowledge to a model because the existing knowledge tends to overwhelm anything you attempt to add in the fine-tuning steps.

Look into different RAG and tool usage mechanisms instead. You might even be able to get good results from dumping large amounts of information into a long context model like Gemini Flash.

First I would calculate the number of tokens you actually need. If its less than 32k there are plenty of ways to pull this off without RAG. If more (millions), you should understand RAG is an approximation technique and results may not be as high quality. If wayyyy more (billions), you might actually want to finetune

No fine-tuning is necessary. You can use something reasonably good at RAG that's small enough to run locally like the Command-R model run by Ollama and a small embedding model like Nomic. There are dozens of simple interfaces that will let you import files to create a RAG knowledgebase to interact with as you describe, AnythingLLM is a popular one. Just point it at your locally-running LLM or tell them to download one using the interface. Behind the scenes they store everything in LanceDB or similar and perform the searching for you when you submit a prompt in the simple chat interface.

Very easy to do with Milvus and LangChain. I built a private slack bot that takes PDFs, chunks it into Milvus using PyMuPDF, the uses LangChain for recall, its surprising good for what your describe and took maybe 2 hours to build and run locally.

Use RAG.

Fine tune will bias something to return specific answers. It's great for tone and classification. It's terrible for information. If you get info out of it, it's because it's a consistent hallucination.

Embeddings will turn the whole thing into a bunch of numbers. So something like Sentinel will probably match with similar feats. Embeddings are perfect for searching. You can convert images and sound to these numbers too.

But these numbers can't be stored in any regular DB. Most of the time it's somewhere in memory, then thrown out. I haven't looked deep into txtai but it looks like what it does. This is okay, but it's a little slow and wasteful as you're running the embeddings each time. So that's what vector DBs are for. But unless you're running this at scale where every cent adds up, you don't really need one.

As for preprocessing, many embedding models are already good enough. I'd say try it first, try different models, then tweak as needed. Generally proprietary models do better than open source, but there's likely an open source one designed for game books, which would do best on an unprocessed D&D book.

However it's likely to be poor at matching pages afaik, unless you attach that info.

All the people saying "don't use fine-tuning" don't realize that most of traditional fine-tuning's issues are due to modifying all of the weights in your model, which causes catastrophic forgetting

There's tons of parameter efficient fine-tuning methods, i.e. lora, "soft prompts", ReFt, etc which are actually good to use alongside RAG and will likely supercharge your solution compared to "simply using RAG". The fewer parameters you modify, the more knowledge is "preserved".

Also, look into the Graph-RAG/Semantic Graph stuff in txtai. As usual, David (author of txtai) was implementing code for things that the market only just now cares about years ago.

RAG is all you need*. This is a pretty DIY setup, but I use a private instance of Dify for this. I have a private Git repository where I commit my "knowledge", a Git hook syncs the changes with the Dify knowledge API, and then I use the Dify API/chat for querying.

*it would probably be better to add a knowledge graph as an extra step, which first tells the system where to search. RAG by itself is pretty bad at summarizing and combining many different docs due to the limited LLM context sizes, and I find that many questions require this global overview. A knowledge graph or other form of index/meta-layer probably solves that.

Don't have anything to add to the others. Just sharing a way of thinking for deciding between RAG and fine-tuning:

(A) RAG is for changing content

(B) fine-tuning is for changing behaviour

(C) see if few shot-learning or prompt engineering is enough before going to (A) or (B)

It's a bit simplistic but I found it helpful so far.

I did something similar to this using RAG except for Vampire rather than D&D. It wasn't overwhelmingly difficult, but I found that the system was quite sensitive to how I chunked up the books. Just letting an automated system prepare the PDFs for me gave very poor results all around. I had to ensure that individual chunks had logical start/end positions, that tables weren't cut off, and so on.

I wouldn't fine-tune, that's too much cost/effort.

You can actually do this with LLMStack (https://github.com/trypromptly/LLMStack) quite easily in a no-code way. Put together a guide to use LLMStack with Ollama last week - https://docs.trypromptly.com/guides/using-llama3-with-ollama for using local models. It lets you load all your files as a datasource and then build a RAG app over it.

For now it still uses openai for embeddings generation by default and we are updating that in the next couple of releases to be able to use a local model for embedding generation before writing to a vector db.

Disclosure: I'm the maintainer of LLMStack project

Thanks for giving txtai a try.

txtai doesn't require Java. It has a text extraction component which can optionally use Apache Tika. Apache Tika is a Java library. Tika can also be spun up as a Docker image much like someone can spin up Ollama for LLM inference.

Looking at your use case, it appears you wanted to parse and index HTML? If so, the only dependency should have been BeautifulSoup4.

Alternatively, one can use another library such as unstructured.io or PyMuPDF for word/pdf. Those are not issue free though. For example, unstructured requires libreoffice for word documents, poppler for pdfs. PyMuPDF is AGPL, which is a non-starter for many. Apache Tika is Apache 2.0, mature and it has robust production-quality support for a lot of formats.

Glad to hear txtai is on your list.

I recently wrote an article (https://medium.com/neuml/vector-search-rag-landscape-a-revie...) comparing txtai with other popular frameworks. I was expecting to find some really interesting and innovative things in the others. But from my perspective I was underwhelmed.

I'm a big fan of simplicity and none of them are following that strategy. Agentic workflows seem like a big fancy term but I don't see the value currently. Things are hard enough as it is.

If your team is already using another framework, I'm sure anything can work. Some of the other projects are VC-backed with larger teams. In some cases, that may be important.

Appreciate the well wishes.

NeuML is not venture backed, so there is no impetus to build a hosted version. The main goal is making it easier for a larger audience.

txtai is not (just) a vector store, it's a full-fledged RAG system. Apples and oranges if you ask me.

Thanks. That is correct. This is an alternative to LangChain/LlamaIndex on the RAG side and Chroma on the vector db side.

We certainly could add typing to the main API calls. Typing isn't a huge thing to me as a developer, so I've never really made it a priority. The only place there is typing is in the FastAPI hooks given it's required.

You can read more on that here: https://neuml.github.io/txtai/embeddings/configuration/vecto...

txtai supports Hugging Face Transformers models, llama.cpp embeddings models and API services such as OpenAI/Cohere/Ollama.

A RTX 3090 is more than enough for 7B LLMs. With 4-bit quantization, you can run inference with an even larger LLM using a 24GB GPU.

If you're using remote API services, you might be able to just use a CPU.

It’s frustrating when people ask for hardware requirements without stating what they are trying to do, do you have 100,000,000 books to index or do you have 5 articles? What are the context lengths you need? What about latency?

How can someone tell you what hardware you need when you give literally no information about what you’re trying to do?

The embedding models at the heart of txtai can be small enough to run on intel CPUs from ten years ago. It's extremely frustrating when HN commentators don't do even the most basic research into the product that they are critiquing.