Talk to Your Data: One Model, Any Relational Database

I worked for Cleargraph, a startup which built a natural language query layer on top of RDBMSes and which we sold to Tableau. We reached the same conclusions as you: that such a system must properly handle ambiguous queries, and users need to explicitly understand the results they're viewing.

The way our system worked, broadly, was with a computational linguistics approach rather than an NLP one. We used an augmented bottom-up parsing algorithm to assemble a user's query into an AST, then translated that AST into both SQL (though that changed post-acquisition) and back into a canonical English-like query.

For ambiguous inputs, the parser would produce multiple candidate ASTs which we could offer to the user in the form of search results, for them to select the expression they preferred (solving your point 1).

The parser was constructed to use semantic information about the database schema to infer fields in the case of underspecification (handling your point 2). For example, we might understand about the `Series` table that `name` was the "display column", and should be the default field by which record-level results were displayed. Semantic information could, to various degrees, be inferred - always with the option to be manually overridden.

It seemed clear to us that a schema-agnostic approach to SQL generation would be untenable for a real product. Annotation of the schema over time (starting with basic information like field names, synonyms, and types, then adding semantic information like currency or unit labels) is key to letting a query generator produce sensible results to underspecified queries. For this reason alone, I'm skeptical of the quality of results that something like Photon can produce. Every schema differs in syntax and semantics. Perhaps a big enough corpus of schemas can allow for a generalized model. The more you can constrain the domain in building such a system, the better the results. (look at Statmuse [0]: pretty impressive query capability over sports facts).

[0]: https://www.statmuse.com/

You're right! I package public trained models for natural language queries (https://github.com/paulfitz/mlsql/), and the models are bad at saying they don't know. I'm optimistic though. There's significant year-on-year improvement (driven by real progress in NLP), and the training datasets are getting more interesting. There are now conversational datasets (e.g. https://yale-lily.github.io/cosql) where the model is trained to ask follow-up questions, and an explicit goal is "system responses to clarify ambiguous questions, verify returned results, and notify users of unanswerable or unrelated questions". That could be a big win.

We paid a lot of attention to this exact problem at Delver, talking to potential users it was a big concern especially for reporting/BI. We’d never give results until we had a single unambiguous parse of a question, but this required a core that was quite old fashioned NLP, with machine learning only really being used around the edges. So ultimately we’d always generate a series of parse trees with embedded semantics that composed together, but the parser could look at things like external ontologies or word embeddings to try and work out the exact analogy in the domain for something the user said (and confirm with them that the new language was what they meant).

It was reasonably cool but I was a terrible CEO and slowly killed it:

https://youtu.be/a2ufq6CdCYw

And critical! Being able to snag a timespan of data, with point in time for records is the majority of what business users want.

How this works usually depends on your data's provenance. Databases get built with lots of different and variously compatible definitions of "correct", sometimes because RDBMS date and time types are inconsistent, but usually during development. Not all of those definitions have a clean path to UTC offset, for example, if you're comparing data from different time zones.

If someone said they'd made a one size fits all query engine that will talk to any arbitrary database from a range of RDBMS and handle date/time consistently and reliably, I'd be skeptical. Maybe if it's operated by someone who knows what they're doing, but that's not exactly the target audience for natural language SQL queries.

I can't seem to get it to correctly answer any query which references more than a single table.

For instance, "which city has the most farms" yields the somewhat nonsensical "SELECT city.City_ID FROM city GROUP BY city.City_ID ORDER BY COUNT(*) DESC LIMIT 1"

It seems fine for basic queries like "how many total horses are there", but that's basically it.

Hmm. When I ask "What is the LName of all the students" it will only give me the first name instead.