Hacker News Reader

Comment by herdcall | Hacker News Reader

We're using LLMs (OpenAI) to generate SQL queries to search customer data, and the current approach using chat API frequently generates queries using the wrong record/column names. I'm exploring use of fine tuning to improve accuracy on a customer/customer basis to train on their set of data, isn't that a good use case?

View on HN

The question is how successful you'll be. Generally fine-tuning means you're going to drop 10-500k on data labeling and compute costs + 1 science type for 6 months.

This means you're easily looking at a 1 Million dollar project in order to be successful. Even once you're done, the odds of success are mixed - and Claude-3 may beat your fine-tuned model. These economics aren't hard for research shops, but startups are going to struggle with this approach.

I think this is something that sample biasing would work better for, which you could do with local LLM's. For example with ad-llama[1] you would just have a sampler bias like so:

  const knownColumns = ['name', 'email', 'id']

  template`
    SELECT "${a('column name', {
      sampler: bias.accept(oneOf(knownColumns))
    })}"
    FROM "table"
  `

You're able to enforce, at the sampler level, that the output is one of the expected choices.

[1] https://ad-llama.vercel.app/playground/

As other commenter said, fine-tuning is a costly and time-consuming affair. You can use RAG and have a separate namespace[1] for each of your customers in the vector database, so that it only searches through a specified customers' data for relevant context.

[1] https://docs.pinecone.io/docs/namespaces

I don't think so. You don't have enough data for finetuning. How would you even fine tune? If you have less then several thousand examples, I wouldn't even think about finetuning

1. Use GPT-4

2. Clearly specify the column names as part of the prompt and that there are no other columns.

3. Occasionally you may get an error that you have to feed back to GPT-4.

Reply on news.ycombinator.com