We'll make the licensing more visible. Stay tuned in a few minutes.
169 karma · joined January 9, 2019
Getting NL-to-SQL to work at the enterprise grade takes work and a specialized engine / agent to do so (we use fine-tuning together with RAG). I don't believe this will be the objective of OpenAI. Additionally, there are plenty of companies who will want to use a LLM not governed by OpenAI (e.g. another open source model). Dataherald will ultimately allow anyone to swap out the LLM of their choosing. At the moment however GPT4 performs far and above any other model that we've tested.
The dataherald engine is perfect. They are able to use natural language, admittedly with a few iterations, to get the cut of data that they seek. This cuts down hours, or even days from their inquiry.
i should note that our (dataherald's) product only sends schema to the LLM, which is important to note.
And perhaps even more importantly, my experience has been that people ultimately wish to understand trends. In other words, an ad-hoc question that is posed and answered in natural language might ultimately be made into a dashboard that lives on the BI tool (so that trends become clearer).
1- Extract intent and entities from the user prompt (the metric they want, date ranges and location)
2- Query structured data from various sources in our data warehouses
3- Construct the final text and accompanying data visualization.
In the first iterations we trained our model for the extraction step and used templated SQL with variables for entities in step 2. However with the release of GPT-4 and rapid development of LangChain we soon migrated to building on the LangChain framework and relying on GPT-4 to write SQL on the fly. From LangChain we use PromptTemplates, Memory, Example Selectors and Chains heavily. The three steps described above eventually became three chains we built and optimized.
Overall we love Langchain. It allowed us to release quickly and leverage abstractions which cleaned up our code. One of the best parts about building on LangChain is the active community on Discord. However it is a sprawling framework and some components still do not seem ready for use in production. For example, we gave up on using SQLAgent pretty early on since it failed some easy testing.
Some of the other challenges we faced: - GPT-4 is very slow and in our experience GPT3.5–turbo does not work well for building complex SQL queries on the fly. As a result the three chains currently take an average of 27s to produce an answer unless it is already cached.
- When working with SQL results and structured datasets, you quickly hit the token limit when passing the SQL results to the LLM. We did not have access to the 32K context window, so once that is rolled out this should be less of a concern.
- How to test? There does not seem to be many QA tools for LLM applications. We ended up building our own solutions to score generated content and flag low scores so we can add examples to our prompts.
Looking forward to the next few weeks, we plan on delivering the following:
1- Templates that allow users to create content that is optimized for blog posts and social media posts out of the box.
2- Fix some limitations in prompts we can handle. You currently can’t compare across geo types (city with national trends).
3- Add more datasets covering neighborhood level data.
4- Add more chart and map types and viz customization features
Would love to hear the community's thoughts. Are there tools or approaches that we should have considered? Does anyone use Langchain’s SQLAgent in production? Know any real estate professionals we should talk to?