This feels like one of those spaces where everyone says LLMs and AI are just going to make it a solved problem, but no-one in the comments here is even mentioning it - maybe not a good problem for LLMs after all?
The post has an entire section discussing this.
The problem with text-to-sql is that, as the post elaborates, writing SQL is not the problem. It's understanding the context and the data:
> On the other hand, a technical person would notice that the question doesn't make sense, and they would ask for more context. They would ask for details about the business person's hypothesis and the problem at hand. Then, they would explain what type of data is available, and work with the business person to formulate a precise and useful question.
Text-to-sql in practice is a solution that nobody was asking for, despite the insane number of SV startups shipping GPT-text-to-sql wrappers as products.
There certainly is like places where LLMs can help (post touches on this briefly), and that is in semantically exploring databases/tables/etc and contexts around data, but this is a very different project and would require a lot of curation from data teams to make it happen.
What we've found to actually work at Definite (I'm the founder) is text-to-semantic-query. This is an older video, but here's an example: https://www.youtube.com/watch?v=44mhLgUYOp8
1. You also have complete control over what the LLM can do / access thru the semantic layer (e.g. you can remove tables that the LLM shouldn't consider for analytical questions).
2. One of the biggest choke points for text-to-sql is constructing joins. All the joins are already built into the semantic layer.
3. Calculating metrics / measures is handled in the semantic layer instead of on the fly with SQL (e.g. if you ask something like "how much revenue did we generate from product X", you wouldn't want the LLM to come up with a calculation for revenue on the fly. Instead, revenue is clearly defined in the semantic layer).
4. The query format for our semantic layer (we use cube.dev) is JSON, which is much easier to control then free form SQL.
The semantic layer gives the LLM a well defined and constrained space to operate within whereas there are hundreds of ways for it to fail writing raw SQL.