aazo11

The agent is LLM agnostic and you can use it with OpenAI or self-hosted LLMs. For self hosted LLM we have benchmarked performance with Mixtral for tool selection and CodeLlama for code generation.

The agent currently executed the generated SQL (limited to 10 rows) and recovers from errors.

Yes. There is a schema linking step which identifies relevant columns and tables including any foreign key relationships if they exist.

The agent also can be finetuned on sample NL <> SQL pairs or they can be used in few shot prompting.

The entirety of the codebase is now open source.

Hi -- the license is Apache 2.0

We use Stainless. Great team and product.

Companies have invested a lot on data tools and data stack. It seems the ROI is either not there or hard to measure and budgets are being cut, or being repurposed to collecting data for LLM training/fine-tuning.

I work in the field and have read a good number of papers on the topic. All existing approaches send DDL commands, semantic layer data, few shot samples etc. This one actually sends a version of the DB. First time I have seen this approach.

Anyone tried this yet or have thoughts?

While the agent does execute the query to recover from errors, the SQLalchemy call execution is limited to a few rows only so if the server is locked there is probably something else going on ;)

We do store golden sql, query history and schemas. Of course everything is encrypted at rest and in transit.

Glad you have been liking it. Feel free to reach out at amir (at) dataherald.com if you need any additional help setting up.

All DML commands are blocked by the engine. You can wrap the returned SQL in a CTE only passing the rows the customer is allowed to access.

Hi -- we do use Fine-tuning together with RAG. To get best in class performance for NL-to-SQL you definitely need to combine both. The good folk at OpenAI dove into this during the last dev-day https://youtu.be/ahnGLM-RC1Y?si=7fv_JTScpBR9lK1R&t=2370

The largest we have successfully deployed is on the OSQuery schema https://osquery.io/ which is 277 tables and lots of business context (malwares, vulnerabilities, Windows registry keys, etc).

For large databases, LLMs do not perform well if you pass the entire schema (either run into context window issues or confuse the LLM with too much info). There is a schema linking step that identifies the relevant schema and only passes that. The schema linking is also done in the fine-tuning process.

How does this compare with LlamaIndex + Langchain SQL agents? It seems these tools still are not ready for production use cases, especially since the data gets sent to OpenAI.

Open source LLMs are catching up with OpenAI. They are also considerably cheaper in the long run. We took a look at how they performed in terms of accuracy vs. OpenAI in an NL-to-SQL workload.

tldr -> For zero-shot prompting, the closed source LLMs are still far ahead in terms of generating accurate SQL. However the gap is closing. With the right dataset for fine-tuning the open-source LLMs might be able to perform at the same level.

Is langsmith still useful if your app is not built with Langchain? Given how early these technologies wouldn't it be better to use a framework agnostic tool to not be tied in to a single framework?

We have been using Langchain since the beginning. We recently deployed Langsmith and it has helped us a lot with monitoring our app in production. Specifically it helped uncover a very nasty bug which was driving up our LLM costs.

Question answering with an LLM on relational data is hard because the LLM does not know the business and data context. Current approaches are to use few-shot prompting and instructions.

This is a sample set up showing how to set up with US real estate data. With ~60 sample NL <> SQL pairs and ~5 instructions the accuracy gets pretty good. The agent does take ~90 seconds to answer each question though.

The mathematical representation of algorithms is now pretty standard undergrad stuff. Why is relational algebra not better known?

Same here. I hit "run" but get an error

Most RAG approaches use a vectorDB and embeddings for schema linking. In this case the fine-tuning is handling schema linking and there is no vectorDB.

In theory one could create domain specific (or industry specific) templates for data. However coming up with a universal structure might be challenging since data is so varied.

Since the issue is often the context, plugging in data dictionaries (and passing those to the LLM) can help

A tutorial on how to fine-tune a GPT3.5 model for Natural Language to SQL tasks and a comparison of its performance vs Retrieval Augmented Generation.

Based on the results, fine-tuning can match and outperform RAG (the approach matches the state of the art on accuracy while being far faster and cheaper). The big challenge for fine-tuning tasks like this is building the training datasets. Things should get even more interesting when GPT-4 is opened for fine-tuning.

Hi -- the /question endpoint does return the generated SQL under the sql_query field of the JSON response: https://dataherald.readthedocs.io/en/latest/api.question.htm...

You can even see the entire chain of LLM calls in the intermediate_steps field to help debug the agent.

Hi -- thanks for the kind words. While right now the engine only works with OpenAI, swapping out other LLMs simply from the envars is on the roadmap. Since we use Langchain for LLM calls this logic is already abstracted away so it will be a small code change.

We have some samples (including a quickstart in the README and docs https://dataherald.readthedocs.io/en/latest/quickstart.html) but will be adding more tutorials in the coming days.

Steve from Hegel-AI had showed us Vanna. Looks super cool. Let us know if you think there is room to collaborate.

Will shoot you an email to connect and chat further.

Thanks for the feedback regarding the Readme. We have some additional detail in the docs here https://dataherald.readthedocs.io/en/latest/index.html (including an architecture doc), but will definitely be incorporating some of the feedback as we iterate on the documentation which.

You can connect the hosted version to your data warehouse using the SQLAlchemy connectors which are in the open source engine, and the connection info will be stored after being encrypted. An on-prem version of the enterprise product is also on the roadmap.

For protecting against errors the approach is to allow for a human to be in the loop as you alluded to. One of the core pieces of the engine is the Evaluation module which combines white box and black box techniques. You can block the response from going back to the question asker if the confidence level is below a certain threshold until a human verifies the response manually (or always put the human from the data team in the loop).