How successful was this effort? How did you know how successful it was?
The results were pretty good: https://gist.github.com/gaurav274/506337fa51f4df192de78d1280...
Another interesting aspect was the money spent on LLMs. We could have directly used GPT-4 to generate the "golden" table; however, it's a bit expensive — costing $60 to process the information of 1000 users. To maintain accuracy while reducing costs significantly, we set up an LLM model cascade in the EvaDB query, running GPT-3.5 before GPT-4, leading to a 11x cost reduction ($5.5).
Query 1: https://github.com/pchunduri6/stargazers-reloaded/blob/228e8...
Query 2: https://github.com/pchunduri6/stargazers-reloaded/blob/228e8...
Are you measuring accuracy with data wrangling prompts? Would love to learn more about that.
I'm skeptical of any claim that "A works better than B" without some numbers to back it up.