LLMs cannot do this. There are many types of human failures, but we somewhat know the parameters and context of those failures. Political/emotional/fear domains etc have their own issues, but we are aware of them.
However, LLMs cannot perform purely objective tasks like simple math reliably.
I’m not confident that this is so. Adult literacy surveys (see e.g. https://nces.ed.gov/use-work/resource-library/report/statist...) consistently show that most people can’t reliably interpret the meaning of complex or unfamiliar text. It wouldn’t surprise me at all if RFK Jr. is antivax because he misunderstands all the information he sees about the benefits of vaccines.
Profitability is not the absolute measure of what should exist.
Perhaps you're thinking that profit-chasing is the only reason companies don't offer good customer support today? That's not accurate. Providing enough smart, well-resourced human beings to answer every question your customers can come up with is a huge operational challenge, unless your product is absolutely dead simple or you're small enough to make random employees help in their spare time.
Valuable to whom? How so?
At that rate I would end my business with such a company.
If you claim such a terrible level of costumer support is valuable, I question your judgement of value.
But LLMs fail the most basic tests of understanding that don't require complexity. They have read everything that exists. What would even be considered unfamiliar in that context?
> RFK Jr. is antivax because he misunderstands all the information he sees about the benefits of vaccines.
These are areas where information can be contradictory. Even this statement is questionable in its most literal interpretation. Has he made such a statement? Is that a correct interpretation of his position?
The errors we are criticizing in LLMs are not areas of conflicting information or difficult to discern truths. We are told LLMs are operating at PhD level. Yet, when asked to perform simpler everyday tasks, they often fail in ways no human normally would.
Which basic tests of understanding do state-of-the-art LLMs fail? Perhaps there's something I don't know here, but in my experience they seem to have basic understanding, and I routinely see people claim LLMs can't do things they can in fact do.
It is an example that shows the difference between understanding and patterns. No model actually understands the most fundamental concept of length.
LLMs can seem to do almost anything for which there are sufficient patterns to train on. However, there aren't infinite patterns available to train on. So, edge cases are everywhere. Such as this one.
Yes, outside the training patterns is the point. I have no doubt if you trained LLMs on this type of pattern with millions of examples it could get the answers reliably.
The whole point is that humans do not need data training. They understand such concepts from one example.