Heh, you seem to keep asking :) You could also ask in our community Discord, tbh, there are people who have been trying both.. There's definitely a ton of great things about langchain, so I'd be curious myself!
To be precise - I don't think I'm saying 'local LLMs' above :) But technically possible, I guess, just hasn't been part of what's officially available. (There are also licensing issues still.) To answer your question about the APIs - the Agent itself queries OpenAI via REST to break the prompt down into tasks, then works with the underlying tools/pipelines using Python API (and then, e.g., a Transformer model that's part of the pipeline has to be 'loaded' into a GPU). Part of those pipelines might be using Promptnode (that can work with hosted LLMs via REST, but could also work with a local LLM). Re 'subsume' - well, that depends :) But arguably, you can build an NLP Python backend with Haystack only, of course.. Regardless of how complex your underlying use case is, or whether it's extractive, generative or both.
Thanks :) Answered a similar one somewhere else here - looks like you've found it already. Feel free to ask more in Discord https://haystack.deepset.ai/community
Thanks :) Working on it. Re local models - indeed, all started with using the Transformer models for extractive QA and semantic search. With the Promptnode, and/or the Agents it's also now possible to combine local models/pipelines & 'LLMs' freely.
Very accurate observation :) So basically, a bit more freedom in picking the right tools for the job, connecting an LLM to proprietary data in a safe way, using multiple models simultaneously, and leveraging custom extractive/generative pipelines.
See above - Haystack started a few years ago as a result of us working with some large enterprise clients on implementing extractive QA at scale. Now evolving to also allow the backend builders to mimic what's available from, e.g. OpenAI+plugins, but with their own set of models, and being able to mix&match best available components and technology.
Haystack has been around for a while now, and we've been mostly specializing in the extractive QA. The focus has been indeed on making the use of local Transformer models most easy and convenient for a backend application builder. You can build very reliable and sometimes quite elaborate NLP pipelines with Haystack (e.g., extractive or generative QA, summarization, document similarity, semantic search, FAQ-style search, etc. etc.) with either Transformer models, LLMs, or both. With the Agents you can also put an Agent on top of your pipelines and use a prompt-defined control to find the best underlying tool and pipeline for the task. Haystack has always included all the necessary 'infrastructure' components - pre-processing, indexing, several document stores to choose from (ES/OS, Pinecone, Weavite, Milvus, now Qdrant, etc.) and the means to evaluate and fine-tune Transformer models.
With a real-life application it's often about making the LLM work on top of your actual (private) data most reliably. By definition a proprietary hosted LLM can't know about it unless you bridge it somehow in a reliable manner.
Thanks for the spotlight :) We've spent quite a lot of time working on the Agents lately, and it's definitely a big focus. Couple of extra points to reflect on some of the comments here. It's quite straightforward to build a hybrid NLP backend with Haystack combining either hosted LLM (e.g., OpenAI or Cohere), or local, smaller Transformer models, or both. Agents add another level of control on top of that, as described in the article and in the comments. This provides more flexibility wrt bridging it to the relevant data and extract/generate accurate non-hallucinatory answers. Join our Discord too :) https://haystack.deepset.ai/community
Re structured data - in theory, yes :) We have to work a bit more in that direction. Here's the first step - querying table data, which could be really helpful for reports, financial data, etc. In regards to the storage backend - it's currently Elasticsearch, OpenSearch, SQL+FAISS/Milvus/Weaviate (when using dense vectors/dense passage retrieval). There is also an in-memory datastore using python primitives for fast prototyping.
Yep, that's definitely this challenge with commonly available models. In a real-life product development there's most often an important step of evaluating the model(s) and fine-tuning if necessary.
Fair enough. That's the inherent 'feature' of the Transformer models, though. We have a series of articles on optimization here https://medium.com/deepset-ai/accelerate-your-qa-system-with... - but also, no, double digits seconds is not necessarily the best result :) Our users have done much better <g>