Even w/ a 20% failure it's better than not having the classifications
It doesn't matter whether that's reasonable or not, there are a lot of people who expect software systems to be totally reliable at what they do, and don't want to accept less.
https://gist.github.com/SMUsamaShah/20f24e80cfe962d26af5315e...
I think that even the simple LLM's are very well suited for classification-tasks, where very little prompting is needed.
So you had a list of products (what sort - I am thinking like widgets from a wholesaler and you want to have a three tier menu for an e-commerce site?)
I am guessing each product has a description - like from Amazon, and chatgpt read the description and said “aha this is a Television/LCD/50inch or Underwear/flimsy/bra
I assume you sent in 200,000 different queries - but how did you get it to return three tiers? (Maybe I need to read one of those “become a ChatGPt expert” blogs
You could plainly ask the LLM something like this as the query goes on:
"Please provide 3 categories that this product could exist under, with increasing specificity in the following format:
{
"broad category": "a broad category that would encompass this product, as well as others, for example 'televisions' for a 50" OLED LG with Roku integration",
"category": "a narrower category that describes this product more aggressively, for example 'Smart Televisions'",
"narrow category": "an even narrower category that describes this product and its direct competitors, for example OLED Smart televisions"
}
"Given a product categorization of a specific level of specificity, please provide a list of words and phrases that mean the same thing".
In OpenAI's backend - and many of them, I think, you can have the api run the query multiple times and get back multiple answers. enumerate over those answers, build the graph, and you can have all that data in an easy to read and follow format!
It might not be perfect, but it should be pretty good!
Text similarity works well in this case. You can just use cosine similarity and merge ones that are very close or ask GPT to compare for those on the edge
Unless you need to have some "reasoning" to classify the documents correctly, a much more lightweight BERT-like model (RoBERTa or DistilBERT) will perform on par in accuracy while being a lot faster.
LLMs can operate as a very, very *very* approachable natural language processing model without needing to know all the gritty details of NLP.
If you already have the answers to verify the LLM output against why not just use those to begin with?
Thinking being that you'd compare outputs of the two, and under assumption of the results being statistically independent from each other and of similar quality, say 1% difference between the two in said comparison, would suggest ~ 0.5% error rate from "ground truth".