Isn’t this relatively trivial to correct? Just like chain of thought reasoning replaces end tokens with “hmm” to continue the thought can’t users just replace the llm tokens whenever it starts saying “maybe they are referring to” with something like. “Let me ask a clarifying question before I proceed.”
Can you clarify what you mean by using an LLM to “get a list that would even out the numbers”? If you’re doing binary classification, you need datapoints for features as well as the target class so how does an LLM synthetically create that without causing problems?
It’s kinda easy to check it though. Ask your friends if they use it, preferably non developer. When I find out my accounting friends and marketing friends are using it for basic analytics/promotion write ups I realized this is being used by non developers already on a regular basis. The use by non tech folks is significant for this . Crypto as a counter example of hype hasn’t lived up to its purported use case as non tech ppl I know only “own” it through an exchange.