Comment by bob1029 | Hacker News Reader

Comment by bob1029 | Hacker News Reader

We tried this for a while and stopped because it seemed like a dead end.

Our use case was/is a text-to-sql bot that would provide domain-specific output. Our domain is very complex so any sort of off-the-shelf AI SQL helper is a joke to us at best. My thinking goes something like "If your schema is so simple that you don't need to think about FTing the model, why do you need its help writing SQL in the first place? Your answers are probably on google somewhere."

The perspective we have now is that the LLM is a probabilistic text transformation engine that must always be contextualized by some external means. Clearly, if we could just include all of our context & it fits & it's cheap, we should just do this. But, the reason anyone is thinking about FT in the first place is because you can't fit the whole damn business in the prompt (or because it's too expensive to do it at scale).

The approach we are looking into now is a classification front end that attempts to discover the relevant business context being referred to by the user's initial prompt. Once we detect our classes, we look up the boilerplate text and incorporate it into the final prompt.

So, instead of trying to align the Jupiter-scale model to your business needs, leave it alone and build a smaller adapter layer that can work with any unmodified LLM.

I just finished augmenting some of the Spider and WikiSQL data on huggingface [1] I initially intended to train a text-to-sql LLM that would take a natural language question and be provided the CREATE TABLE statements to provide some grounding for the response. So instead of hallucinating the column and table names by using the question alone, I was hoping the CREATE TABLE statement would limit the choices. We'll see if it's useful or not, but funny enough I came across this article after I finished the dataset.[2]

I'd definitely like to see how others are doing it.

[1] https://huggingface.co/datasets/b-mc2/sql-create-context

[2] https://blog.langchain.dev/llms-and-sql/

I am currently taking a similar approach where we use an embedding and vector query to create the context relevant to the question the user posed and then throw it to the LLM. It's a hit or miss, esp for look ups of relevant code literals.

Been working on the same thing. In my case, I generated synthetic data based on the API’s schema and trained a T5 on that. Basically treated it as a translation problem from human language to API.

Are you documenting or sharing your progress anywhere? This exactly describes something I've been asked to investigate.

I can't share anything much more specific than what I already have, unfortunately.

All I can really add - Binary classification is like a super power once you understand the statistics around it. If you can find clever ways to combine multiple binary classifiers, you can quickly narrow down relevant context. You can also use the statistics to do things like determine if a query is too vague to be serviced by an LLM backend. You can also answer questions like "Why didn't we consider a table or join when writing the user's query?".

Reply on news.ycombinator.com