yuhongsun

RAG is a tool for the deep research agent to use in finding all of the context it needs. Deep research can call the search many times and reflect on the results of the previous searches and then search for other things as needed. Deep research flows can also generate chain-of-thought type outputs that is not for searching or for directly answering the user.

Yup, hopefully with Onyx, the folks who have these questions can just fire off a query with agent mode turned on and the LLM will research the relevant tree of knowledge and come back with an answer in the fraction of the time it would take for people to do it with all of the handoffs in between.

We have a dataset that we use internally to evaluate our search quality. It's more representative of our use case since it contains Slack messages, call transcripts, very technical design docs, company policies which is pretty different from what embedding models are typically trained on.

We checked the recall at 4K tokens (which was a pretty typical token limit of the previous generation of LLMs) and we were at over 94% recall for our 10K document set. We also added a lot of noise to it (Slack messages from public Slack workspaces) to get hundreds of thousands of documents but recall remained at over 90%.

Assuming self-hosting, data is processed within the deployment with local deep learning models for embedding, identifying low information documents, etc. A hybrid keyword/vector index is built locally within the deployment as well.

At rest, the data is stored in Postgres and Vespa (the hybrid index), both of which are part of the deployment so it's all local.

The part that typically goes external is the LLM but many teams also host local LLMs to use with Onyx. In either case, the LLM is not being finetuned, the knowledge relevant to the question is passed in as part of the user message.

We built Onyx with data security in mind so we're very proud of the way the data flows within the system. We made the system work well with models that can run without GPUs as well so our users can get good quality results even if deploying on a laptop.

This is a large challenge in itself actually. Every external tool has it's own framework for permissions (necessarily so).

For example, Google Drive docs have permissions like "global public", "domain public", "private" where "private" is shared with users and groups and there's also the document owner.

Slack has public channels, private channels, DMs, group DMs.

So we need to map these external objects and their external users/groups into a unified representation within Onyx.

Then there are additional challenges like rate limiting so we cannot poll at subsecond intervals.

The way that we do it is we have async jobs that check for object permission updates and group/user updates against the external sources at a configurable frequency (with defaults that depend on the external source type).

Of course, always failing-closed instead of failing-open and defaulting to least permissive.

It's like how OpenAI's deep research works by searching the internet, ours works by searching over our "RAG" system that indexes company documents.

Quite a lot to cover here! So in addition to the typical RAG pipeline, we have many other signals like learning from user feedback, time based weighting, metadata handling, weighting between title/content, and different custom deep learning models that run at inference and indexing time all to help the retrieval. But this is all part of the RAG component.

The agent part is the loop of running the LLM over RAG system and letting it decide which questions it wants to explore more (some similarities to retry|refuse|respond I guess?). We also have the model do CoT over its own results including over the subquestions it generates.

Essentially it is the deep research paradigm with some more parallelism and a document index backing it.

How does the agent traverse the information: there are index-free approaches where the LLM has to use the searches of the tools. This gives worse results than approaches that build a coherent index across sources. We use the latter approach. So the search occurs over our index which is a central place for all the knowledge across all connected tools.

Do you have any internal evals on how well the different model affect the overall quality of output, esp for a "deep search" type of task? I have model-picker fatigue: Yes, we have datasets that we use internally. It comprises of "company type data" rather than "web type" data (like short Slack messages, very technical design documents, etc.) comprising about 10K documents and 500 questions.

For which model to use: it was developed primarily against gpt-4o but we retuned the prompts to work with all the recent models like Claude 3.5, Gemini, Deepseek, etc.

Do you plan to implement knowledge graphs in the future? Yes! We're looking into customizing LLM based knowledge graphs like LightGraphRAG (inspired by, but not the same).

On privacy and security, we are the only option (as far as I know) that you can connect up to all your company internal docs and have it be all processed locally to the deployment and stored at rest within the deployment.

So basically you can have it completely airgapped from the outside world, the only tough part is the local LLM but there are lots of options for that these days.

Amazing to hear from a happy user! Thanks for the kind words!

Before sharing how it works, I want to highlight some of the challenges of a system like this. Unlike deep research over the internet, LLMs aren’t able to easily leverage the built in searches of these SaaS applications. They each have different ways of searching for things, many do not have strong search capabilities, or they rely on their internal query language. There are also a ton of other signals that web search engines use that aren’t available natively in the tools. Examples include: backlinks, clickthrough rates, etc. Additionally a lot of teams rely on internal terminology that is unique to them and hard for the LLM to search for. There’s also the challenge of unifying the objects across all of the apps into a plaintext representation that works for the LLM.

The best way we’ve found to do this is to build a document index instead of relying on application native searches at query time. The document index is a hybrid index of keyword frequencies and vectors. The keyword component addresses issues like team specific terminology and the vector component allows for natural language queries and non-exact matching. Since all of the documents across the sources are processed prior to query time, inference is fast and all of the documents have already been mapped to an LLM friendly representation.

There are also other signals that we can take into account which are applied across all of the sources. For example, the time that a document was last updated is used to prioritize more recent documents. We also have models that run at indexing time to label documents and models that run at inference time to dynamically change the weights of the search function parameters.

Hey everyone, I’m one of the authors of the post. As an engineer at small companies, I’ve only ever heard about ERPs but never fully understood them. They seemed like clunky, expensive software that only large, non-technical companies use.

Lately, as my startup has started serving more enterprise customers, ERPs have come up more frequently. So, we decided to dive in and learn more about them to better serve our clients. It was interesting to learn that both SAP and MRP II spawned out of IBM!

Despite all the research, I realized I’m the kind of person who only truly understands something by getting hands-on. I guess as a next step I’ll have to try a pseudo-setup of an ERP myself. :)

Danswer AI | https://www.danswer.ai/ | Full Stack Engineer, Customer Engineer | On-site | $150,000 - $240,000

Danswer is an open-source GenAI assistant with organization specific knowledge. Danswer is focused on connecting all of a businesses disparate knowledge in a single place, enabling teammates to find answers instantly. We are used by fortune 500 companies and we’d love to share more details over an introductory call We came out of the most recent YC batch (W24). We have been seeing incredible demand and we are looking to expand our team to keep up with the interest.

Check out our Github with 10,000 stars here: https://github.com/danswer-ai/danswer

Check out our job postings here: https://www.workatastartup.com/companies/danswer

Feel free to email your resume or refer somebody to founders@danswer.ai

We have a connector interface and build guide for contributors: https://github.com/danswer-ai/danswer/blob/main/backend/dans...

Should be not too bad to build one out! Fun fact, more than half the connectors were built entirely by community members who needed them for their own teams and we're super grateful when they contribute it back to the repo.

Ya, I haven't dug too deep into their project, to be honest. But we do think it's great more teams are going for open source.

I tried to look up their RAG pipeline to see if they've invested effort into building a strong retrieval. It looks like they have a more basic vector only search, not sure if it's running local models or using a cloud service. On the flip side, it looks like they have assistants that can take actions (which we don't do), so maybe that's their focus.

I can only speculate as I've never spoken with their team but I imagine they're working closely with a few specific teams judging based on their number of connectors. Most likely it was built for specific customers. We are leaning more into the open source and working with our community and letting people self-host easily to find value with no friction.

There's a question by cpach further up on this page which is essentially asking this but also with some additional questions. Hopefully you find that thread useful!

TLDR: There's the cloud version and also there are a set of nice to have features more relevant for larger teams. Those features are not MIT.

Are you referring to the approach of creating hypothetical questions for embedding along with the document. Or more along the lines of creating summaries of the documents during indexing and embedding those as well?

Either case, the reason we don't do LLM based transformations during indexing time is that we hope our tool can be used by teams with a lot of documents. It becomes prohibitively expensive to run these transformations at indexing time when the scale can be multiple millions of documents.

Two other tidbits on this:

1. There's a difference between relevance and usefulness that cross-encoders cannot capture. Imagine a thread with a bunch of people complaining about an exception and each comment is another mention of the exception. Now imagine another thread with one mention of the exception at the top, and a bunch of people offering solutions. If you query for the exception, LLMs will find the second thread more useful, but cross encoders will find the first one more relevant.

2. LLMs/GenAI models don't output a single value. They can use the tokens they output to "reason" about the usefulness of a doc. Eg. Rerankers are like tiny LLMs that are only allowed to output "yes" or "no", but instead you can use an LLM to do chain-of-thought and finally decide at the end.

Hi, it may be an indexing issue. There's a precanned "Information not found" message that we show if document retrieval failed. A couple common causes for this are:

- Not provisioning enough resources and processes are dying (we run NLP models locally so the system isn't totally lightweight) - Access is not correctly configured, either while pulling in from the source or at the user level during query time.

Assuming nothing is wrong with the setup, it is most likely because of how we prompt the LLM. Questions that require reasoning or are more open ended, generally aren't "safe" for the LLM to answer. So the system is constrained on purpose. For example if someone were to ask: "How can I increase revenue by 30% next quarter". It's not safe for the system to just propose some actions and it's likely better to just search the documents and say there wasn't any answer in the docs (unless of course some doc explicitly states plans for increasing revenue).

Thanks for the kind words! There's nothing more motivating in the world than hearing that our users are loving what we've built!

Thanks for the kind words! Sorry for the delayed response, this post drew a lot more interest than anticipated and we've been swamped working with new folks coming in.

Regarding:

- PrivateGPT: they're for individual use where you ingest your own data. We're for teams to use with access controls, connectors to typical business SaaS tools, works at scale (incremental updates and scalable container architecture), different user roles, etc. So basically I see very little overlap between the two projects at least in terms of "competition", we're just different.

- LlamaHub: Our connectors pull all of the documents in the first run, then every following run, it only pulls in documents that have changed since the last run. For large teams, the first run may take many hours but following that, it will only take seconds each time. Without this, it becomes untenable to keep all information up to date. Also we pull in additional metadata and permissions, which not all LlamaHub connectors support.

- Push flow for indexing: Yes, we also have APIs that you can push to for indexing documents. For event based "push", we didn't go that route because most tools we connect to don't support this.

- Knowledge Graph: We will certainly be building this, it is only a matter of prioritization and timelines.

- Replacing the custom search: We think our search is much better than a basic RAG pipeline out of the box that someone can get from Langchain/LlamaIndex. What's the motivation for wanting to remove/replace it?

Ya, a lot of knowledge is locked away in metadata, how documents are structured, in non-textual representations, or sometimes even just in the head of experts. We definitely want to be more than a shallow access to documents and we're building with that in mind. We're currently working to include more metadata and organizational understanding, with plans to tackle OCR, NL-to-SQL, code search and knowledge graphs in the future. While we don't have concrete dates for all of these, hopefully this gives some visibility into our vision at least!

Regarding the context size, the research community is doing some stunning work. If you're interested, you should check out the new Mamba (Transformers replacement) architecture: https://arxiv.org/abs/2312.00752

Ah thanks for pointing it out, it look like an awesome project! We love open source so it's nice to see others going this direction as well

I think the explosion in interest and all the new software around GenAI owe their success to NLP advancements (coming from an NLP background myself, I may be biased though). I think the best projects are those pushing the frontier of NLP and sharing the learnings with the world:

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 - Mixture of Experts model like GPT-4 but this one is open source doesn't cost an arm and leg to run.

https://huggingface.co/mixedbread-ai/mxbai-rerank-xsmall-v1 - This is a very recent reranker, we will probably try it out. It's really nice that it's open source and small enough to run on CPU. I think an overall trend of the space in the next years will be to maintain/improve performance while shrinking the cost and hardware requirements.

So private refers to two things here, sorry for any confusion.

When we say "chat over private data" we mean that this data isn't publicly available and no LLMs have this knowledge in their training. Meaning that with our system you can now ask questions about team specific knowledge. For example, you can ask questions like "What features did customer X ask about in our last call". Obviously if you ask ChatGPT this, it will have no idea.

The other part is data privacy when using the system. The software can be plugged into most LLM providers or locally running LLMs. So if your team doesn't trust OpenAI but instead has a relationship with say Azure, or GCP, you can just plug into one of those instead. Alternatively, a lot of users recently have been setting up Danswer with locally running LLMs with tools like Ollama. In that case, you now have a truely airgapped system where no data is ever going outwards.

Access controls is a non-glamorous but critical piece of what we're building. Currently implementing automatic access sync-ing for a few sources like Google Drive, Confluence, Jira, and Notion to start. By matching document-access in the source to users and groups, and then to emails, we can finally map Danswer users to document level access. So someone searching in Danswer will only get results based on the set of documents they have access to in the source tool.

For Slack it would look something like: get the users in the Slack channel, map those Slack users to users in Danswer. Then only those users in Danswer will be able to get results from that channel.

Being able to filter down the data deterministically is a big value add, especially as the number of documents scale into the range of multiple millions. We have filters by document-set, tags, time range, source type (ie. only include Slack + Google Drive, or Confluence + Jira + Gong, etc.)

The challenge is with the non-deterministic portions of the flow as you pointed out. Ensuring retrieval quality in out-of-domain datasets, guardrailing the LLM generation, working with conflicting or deprecated information are some of the interesting areas we're addressing. Happy to dive deeper on any aspect you're curious about, and I'm sure we can learn from the discussion as well.

Exactly, there is a huge amount of value in being able to quickly get a holistic view of topics. Most topics don't exist in an isolated tool - most often there are the official discussions/designs which exist in place, there are customer interactions with the topic which uses a separate channel, and then there are one off small conversations about the topic in chats like Slack. So isolated, tool specific searches are great for finding specific documents, but less useful for getting actionable insights.

Regarding contextualizing: we're currently working on organizational understanding and we're very excited about this one! We're embedding users based on the documents they authored or interacted with, the questions they have asked, description of projects they worked on, and the org chart. The thinking is that, there will always be questions that can't fully be answered via documentation alone. But in those cases we'll be able to recommend someone who might know. It also has the benefit of contextualizing the user asking so that we can surface more relevant results for them.

Thanks so much for sharing! Excited to look into it!

Hello, thanks for the kind words! With regards to your questions:

1. Are you referring to the local NLP models or the LLM? The local models are already open source models or ones we've trained ourselves. If you're talking about the LLM, the default is OpenAI but it's easy to configure other ones without any code changes.

2. Most sources are pulled from every 10 minutes. They have incremental updates so if you have Confluence with a million pages, probably in the last 10 minutes, only a dozen or so have been updated. The only exception is websites (which are crawled recursively so we don't know which pages are updated before we try), which is updated once a day.

3. Glean is indeed similar. Without going into the features in detail, we are an open source Glean with more of an emphasis on LLMs and Chat.

4. There's generally not a great reason to build from scratch if an open source alternative with +75% alignment exists. They can always build on top of us if they want. A lot of teams reach out to us because they were looking to switch from their in house solution to Danswer. Generally though these are larger teams, we haven't seen many SMBs building RAG for their own usage, usually these smaller teams building RAG are looking to productize.

5. Currently there is no cheap and fast way to fine-tune LLMs every time a document is updated. If you want an LLM to remember the document that was just updated you'd have to augment it to at least dozens of similar (but all correct) examples. RAG is still the only viable option. Then there is the problem of security etc. since you can't enforce user roles at the LLM level. So companies that focus on building LLMs don't really compete in this specific space and they don't want to either as they're trying to build AGI. There is more of a threat from teams like Microsoft and Google who are indeed trying to build knowledge assistants for their product lines, but we think there is a world where open source ends up winning against the giants!

So one of the main things we do is automatically sync-ing documents from different sources of knowledge from your team. So all of the data connectors as well as the user authentication and access systems would have to be built from scratch if you did your own. Also if you have more than a few documents you would have to recreate the RAG pipeline (and ours is fairly involved so it would be quite some work). Finally there's the UI and other features like learning from feedback, usage analytics, chat history, etc etc.

If you're just looking to upload a few personal docs into a chat assistant for your own use, probably Danswer is overkill and more complex than the effort is worth. If you're thinking of a team wide use case, then using Danswer makes sense.