Show HN: Data content creation tool using OpenAI+LangChain(lessons learned)

While LLMs like ChatGPT have revolutionized content creation, they still cannot create factual and visually-appealing data-intensive content that many industries such as real estate brokers, consultants or financial advisors need. With Dataherald AI (https://www.dataherald.com/sign-up), we made a technical preview of a tool that can automate the creation of data-intensive content for real estate professionals. Users can enter their queries and questions in a prompt-based interface about real estate market trends and the system returns factual text and accompanying data visualizations. The core engine works in three steps:

1- Extract intent and entities from the user prompt (the metric they want, date ranges and location)

2- Query structured data from various sources in our data warehouses

3- Construct the final text and accompanying data visualization.

In the first iterations we trained our model for the extraction step and used templated SQL with variables for entities in step 2. However with the release of GPT-4 and rapid development of LangChain we soon migrated to building on the LangChain framework and relying on GPT-4 to write SQL on the fly. From LangChain we use PromptTemplates, Memory, Example Selectors and Chains heavily. The three steps described above eventually became three chains we built and optimized.

Overall we love Langchain. It allowed us to release quickly and leverage abstractions which cleaned up our code. One of the best parts about building on LangChain is the active community on Discord. However it is a sprawling framework and some components still do not seem ready for use in production. For example, we gave up on using SQLAgent pretty early on since it failed some easy testing.

Some of the other challenges we faced: - GPT-4 is very slow and in our experience GPT3.5–turbo does not work well for building complex SQL queries on the fly. As a result the three chains currently take an average of 27s to produce an answer unless it is already cached.

- When working with SQL results and structured datasets, you quickly hit the token limit when passing the SQL results to the LLM. We did not have access to the 32K context window, so once that is rolled out this should be less of a concern.

- How to test? There does not seem to be many QA tools for LLM applications. We ended up building our own solutions to score generated content and flag low scores so we can add examples to our prompts.

Looking forward to the next few weeks, we plan on delivering the following:

1- Templates that allow users to create content that is optimized for blog posts and social media posts out of the box.

2- Fix some limitations in prompts we can handle. You currently can’t compare across geo types (city with national trends).

3- Add more datasets covering neighborhood level data.

4- Add more chart and map types and viz customization features

Would love to hear the community's thoughts. Are there tools or approaches that we should have considered? Does anyone use Langchain’s SQLAgent in production? Know any real estate professionals we should talk to?