aazo11

Thanks. Let us know if we can help out in any way. We just set up a Discord server you can join as well https://discord.gg/A59Uxyy2k9

Depends what you mean by 'work.' You can definitely use them, but GPT-4 is the only model that generates good SQL without lots of training + fine-tuning (and it is very difficult/impossible to come up with Nl <> SQL training sets). So GPT-4 is the only model that works well in terms of accuracy for now.

That being said, I am pretty sure the self hosted models will get there very soon.

Are you looking to deploy a solution with a self-hosted model?

Sort of. Having perfect data engineering is a requirement if you want to connect an LLM straight to your data warehouse. For real world scenarios, you need a way to add context over time (including examples of how to answer questions from messy data). The same way a new team member would need to be on-boarded the tool needs to learn the context of the data and business logic, store it under supervision from an admin and be able to retrieve it in generating the SQL.

SQL is definitely an old technology. However it is still the main language for interfacing with structured data, even for newer tools like Clickhouse. Down the road it is conceivable that it will be replaced with something else, but the cost of migrating existing data to those will be so high I imagine no-one will take the step to proactively move.

Amazing! Feel free to shoot me a line at amir at dataherald

We are looking to partner with other open source projects. Let's connect. Shoot me a line at amir at dataherald.com

Interesting. So far we have been focused on allowing non-technical users to self-serve from enterprise data warehouses. It might be possible to enable this scenario with a simple addition to the NL-2-SQL engine to translate the generated SQL to API calls. If that does not work, a custom imeplementation of the NL-2-SQL module would def solve this.

Do you have a specific use-case in mind? Would love to collaborate on adding it to the open-source library if you are open to it.

Totally agree. In the hosted version built on top of the engine we block the answer from going to the question asker until an authorized user 'verifies' the answer. However these 'verified' answers are then stored in the context store and retrieved for few shot prompting.

TLDR -- We currently do not automatically load stored procedures as context for in context learning/prompting but it is on the roadmap.

Additional color: Once you configure a connection to a DB, you can trigger a 'scan' of the DB through the API which detects various tables/columns, identifies low cardinality (categorical) columns and their values and also loads in VIEWS stored on tables. In our default NL2SQL agent, these are then used together with 'verified' NL2SQL pairs and with other metadata you can manually add to generate the SQL.

We are looking to automatically add context from Stored procedures and other sources like dbt and various data dictionaries as well.

Hi -- would love to connect and discuss more. Shoot me an email amir at dataherald.com.

Hi Dylan -- new GPT-4 class LLMs have gotten good at writing correct SQL, but while the SQL they generate almost always executes correctly they often write SQL that generates incorrect answers to the question. Some reasons for this can be the business context or definition is not in the table schema, or if the correct query is complex and requires multiple joins.

This is great. Thanks for the contributions on the LangChain discord as well.

At Dataherald our mission is to make it easy to create data-heavy content. While LLMs like ChatGPT have revolutionized content creation, they still cannot create factual and visually-appealing data-intensive content that many industries such as real estate brokers, consultants or financial advisors need. With Dataherald AI (https://www.dataherald.com/sign-up), we set out to deliver a technical preview of a tool that can automate the creation of data-intensive content for real estate professionals. Users can enter their queries and questions in a prompt-based interface about real estate market trends and the system returns factual text and accompanying data visualizations.

The core engine works in three steps:

1- Extract intent and entities from the user prompt (the metric they want, date ranges and location)

2- Query structured data from various sources in our data warehouses

3- Construct the final text and accompanying data visualization.

In the first iterations we trained our model for the extraction step and used templated SQL with variables for entities in step 2. However with the release of GPT-4 and rapid development of LangChain we soon migrated to building on the LangChain framework and relying on GPT-4 to write SQL on the fly. From LangChain we use PromptTemplates, Memory, Example Selectors and Chains heavily. The three steps described above eventually became three chains we built and optimized.

Overall we love Langchain. It allowed us to release quickly and leverage abstractions which cleaned up our code. One of the best parts about building on LangChain is the active community on Discord. However it is a sprawling framework and some components still do not seem ready for use in production. For example, we gave up on using SQLAgent pretty early on since it failed some easy testing.

Some of the other challenges we faced: - GPT-4 is very slow and in our experience GPT3.5–turbo does not work well for building complex SQL queries on the fly. As a result the three chains currently take an average of 27s to produce an answer unless it is already cached.

- When working with SQL results and structured datasets, you quickly hit the token limit when passing the SQL results to the LLM. We did not have access to the 32K context window, so once that is rolled out this should be less of a concern.

- How to test? There does not seem to be many QA tools for LLM applications. We ended up building our own solutions to score generated content and flag low scores so we can add examples to our prompts.

Looking forward to the next few weeks, we plan on delivering the following:

1- Templates that allow users to create content that is optimized for blog posts and social media posts out of the box.

2- Fix some limitations in prompts we can handle. You currently can’t compare across geo types (city with national trends).

3- Add more datasets covering neighborhood level data.

4- Add more chart and map types and viz customization features

Would love to hear the community's thoughts. Are there tools or approaches that we should have considered? Does anyone use Langchain’s SQLAgent in production? Know any real estate professionals we should talk to?

Programmatic SEO is the practice of creating landing pages at scale to capture long tail search traffic. The solution works especially well if your business deals with lots of data, since you can create unique pages that also update. At Dataherald we tried to build a programmatic SEO for our own data and found it to be far harder than anticipated so we wanted to share our experience.

Why was it hard?

- Unless you are AirBnB or Zillow, you will have a limited crawl budget. To get the most out of your budget with a high number of pages, you need to statically generate every single page in advance. Next.js (Vercel) and other static site generators make this easier than it was 5 years ago, but it is still quite a context switch if your entire front-end is a single page application.

- You can’t just drop a table, graph or a few lines of text on the page. Google will penalize your domain for creating spam, low quality content and your domain will be in the dog-house (bye bye organic traffic). The content has to be high quality and engaging for the visitor.

- If your data points will change over time (i.e the number of pages themselves are dynamic) you will have to automate your sitemap creation and maintenance

- You need data. This was the easiest part for us since collecting and organizing external data was our core business, but building and maintaining data pipelines is not easy in general.

How did we solve it?

- We already have millions of data points from pubic and private sources in our warehouse, so data availability wasn’t an issue

- Vercel on the front-end

- We took model driven approach to the backend architecture to allow us to create various front-end templates (HTML code displaying visuals, text, CTAs, etc) from our various data sources. The templates are JSON schema that define how to generate thousands or millions of landing pages. A service creates the pages and then updates them automatically on data source updates

- We are exploring various structured data to text AI solutions but still doing lots of manual templated text writing with SQL.

- Classic SEO optimizations on the page (Ahrefs research, meta tags, H1 tags, canonical, …)

What do you think?

- If the approach above sounds genius or stupid, we would love to hear from you

- If you know of a good structured data to text AI solution, email us

- If you want to implement programmatic SEO please reach out

Amir here. We allow users to download the cleaned data and load it into any visualization tool they want. That being said, we have found a lot of users (especially those who tell stories with data) have a lot of use for having the visualization as part of the same solution.

Question for the HN community: Is there a dataset (free or proprietary) which you regularly use for as part of your work?

Our eventual goal is to remove the need for anyone to do redundant data engineering work on public datasets. So while Superset/Metabase are tools to explore you enterprise data, we are focused on building a growing library of feeds from public data. That being said, we allow users to add to the library from datasets they can point us to.

Thanks for the feedback on the landing page. We will take it into account in the next iteration of the design.

We do not manually create any graphs. The infrastructure we have developed allows us to create new graphs without writing any code:

- On the backend we can add new data pipelines to any public data by setting a few configuration parameters

- The front end is completely modular with re-usable D3 components

While we are doing the configuration ourselves for now, we plan to open this up over time.

Least favorite would definitely be California campaign finance data.

Currently our data viz get embedded through an iFrame so the backlinks are from our front end domain, unless we get an explicit shoutout in the article. However, we plan on moving the "powered by HiGeorge" HTML tag outside the iFrame to help with SEO.