Absolutely. Give it a try. I pasted these comments into ChatGPT and asked it to score them according to slashdot’s system (which i didn’t actually define) and got this reply:

In Slashdot's scoring system, comments are scored from -1 to 5, based on factors like insightfulness, informativeness, and whether they're interesting or funny. Here are the hypothetical scores for the comments from Hacker News:

owlninja (42 minutes ago): This user is asking about the applicability of LLMs for text categorization in large datasets. It's a straightforward query, indicating a need for information. Score: 3 (Interesting) - As it opens up a discussion on a relevant and technical topic.

causalmodels (18 minutes ago): This comment provides a direct solution with a resource (zod-gpt). Score: 4 (Informative) - It not only addresses the query but also provides a specific tool to help achieve the goal, adding value to the discussion.

quickthrower2 (41 minutes ago): This comment suggests an approach for creating buckets using a sample and an LLM. Score: 3 (Interesting) - It proposes a practical method, contributing constructively to the original query.

These scores are subjective and would depend on the perspectives of the individual moderators or the community's voting on Slashdot.

If you know your buckets you can pretty easily get most LLMs to generate structured outputs using something like zod-gpt [1].

[1] https://github.com/dzhng/zod-gpt

Have a look at Using text embeddings and approximate nearest neighbour search to explore ...

https://medium.com/gft-engineering/using-text-embeddings-and...

You can reduce sentences to vectors and then create similarity scores to build a graph over the corpus. If you choose to create clusters then use a llm to summarise them to create labels.

I just did this today using openAI’s function calling. I have a bunch of elements in a scene and trying to classify them between various ‘buckets’ has been the challenge. The way I set it up is that in the schema expected, you take that top level free text, wrap it in quotes and it becomes the parent object with the elementCategory as a string required inside it, with a list of all category types in the description. Then loop through and create a dynamic schema based on how you chunk your data, then add a validation step at the end to ensure that chatgpt doesn’t forget, add or change keys. I found that if you create a numberedKeywordString for your chunk, and wrap each in quotes, then stuff that numberedKeywordString into the system prompt then it’s solid and will ensure the LLM catches every key and gets the associated elementCategory value.

It works quite well for dynamic classification in my purposes but I’m sure there is a better way.

With no problem. The easiest way is to go with GPT4 api for a start (you just tell it how you want it organised / what metadata to extract) - it will be costly, but you can prove that it works, and generate examples to fine tune a simpler model to do the same.

It might help you create the buckets at least if you take a sample of say 1000 and give that to the LLM.

absolutely here is an example with openai. First pass you explain the task and let it reply to a hundredish examples. Fix the errors, fine tune. And it should be much better. You can repeat that any time you need until you reach satisfactory results but they come quickly when I worked on large number of short texts i wanted to classify and extract info from.