alandu

An example question in that eval set is "How many publications were published between 2019 and 2021?". That's something GPT without any context can understand how to answer from a schema (which I assume has a column called publications). An example question that I'd get in my previous role at an ecommerce fraud detection company could be something like "what's the chargeback rate on the ATO segment". Neither chargeback rate nor ATO segment are defined in the database schema. Not only did they have different definitions depending on the context (e.g. which customer), the definition also change over time within the same context.

The bottleneck was mostly in writing the SQL query, which took a lot of time due to the messiness/complexity of the data

Thanks we will rethink how the product is presented on our website!

Right, we are not open source, but the IDE is free to use. The Slack integration and other product offerings that are in the works will be in the 'premium' version of the product that's sold to enterprise

There are other product additions in the works, like hooking it up to your locally opened Slack. The plugin would be limiting

If you open a .sql file into the workspace, the queries in that file will be auto parsed and used as context for Q&A. If you're willing, would love to help debug - could you email support@dalmatian.ai

We have not come across any benchmark dataset that's actually worth evaluating on because the questions are not representative of real world enterprise problems. They don't reflect the degree of context needed to answer domain/business-specific questions accurately.

I used to get slammed with so many requests that my boss had to tell the sales team to reduce the number of questions and only ask highest priority ones. Analytics teams serve a lot of different teams in an org, and the requests can really pile up. I was basically a bottleneck, which was a lose-lose for me since I was slammed with work and for business stakeholders too since they had to either wait a long time for responses or were limited in what they could even ask.

1 - yes our current solution does require you to be allowed to use ChatGPT/OpenAI. Unfortunately the accuracy using smaller models (even GPT-3.5) is poor. We don't see a local model (which will be much worse than GPT-3.5) even with fine tuning being anywhere close to good enough (would also require a really large number of queries). So we are relying on GPT-4 for now.

2 - agreed the background isn't why anyone should adopt a tool, just wanted to share our story. I would add that creating a good wrapper can actually be quite challenging, need to synthesize many pieces under constraints like memory, compute, speed, accuracy.

Just want to clarify that OpenAI does not train on the query code and schema info we send via API. It's equivalent to using https://chat.openai.com/ setting "Improve the model for everyone" (previously "Chat history & training") to off in Data Controls.