It's only a matter of time before it's fully there.
The error is deriving human ideas of 'quality' from that.
The stochastic parrot might well be better at your job than you are.
Obviously what I and everyone means by stochastic parrot is that it's not intelligent. It's wrong. It is intelligent. At worst it's as intelligent as a mentally retarded/schizophrenic or insane human. But even a mentally handicapped human still displays a level of intelligence.
Unfortunately the way LLM's are architected right now means they will never be "fully there".
Sure, those things could make some things easier, but why are those things that humans don't have suddenly "non negotiable" when a machine also can't do them?
Legal issues?
Neither of which are supplied by education, and I am unaware of any humans basing able to deliberately (let alone precisely) alter their own perception on the scale between precision and recall — look at something, it's "obviously" X or not X almost immediately after you know what the category X is, even when you're wrong. Apart from the very first few encounters it doesn't even matter how much of a noob or expert you are in the field of X-recognition, your confidence is the same.
Worse:
> qualified
Given how well ChatGPT does on standardised tests, doing better than many actual humans even despite its many flaws and limitations, it should be clear that the qualifications are not good enough to do what you're expecting them to do.
> and aren't just bullshitting plausible-sounding words.
That's demonstrably how humans work (at least when it can be tested, perhaps people who need split brains are weird): all the indications are we do a thing first and then come up with a justification after.
(And then we have people like Boris Johnson, 2:1 BA from Oxford, with a disconnect between reality and the words leaving his mouth that would be comical except he actually became Prime Minister in real life and not just a TV comedy blending 'Allo 'Allo with The Thick of It).
1) statistical intervals are mathematical artifacts of our techniques that describe the samples observed and trained on. They aren’t ground truth observations of the underlying process or populations. We put too much weight in them.
2) you can absolutely observe precision and recall from online performance and compare that directly against human performance on known labeled data. From that you can determine which has the better error rates. That is entirely sufficient for almost all practical use cases.
3) obviously it would be better if we could derive confidence of a classification, but given the fact LLMs aren’t directly reasoning or optimizing the statistical properties of some mathematical problem they will never have the same character as say regressions or other statistical techniques that are some form of mathematical optimizer. They’re just solving the problem in fundamentally different ways.
4) it’s not clear to me statistical optimization is a universally superior technique. The reality is many problems are better solved with an abductive reasoning technique like LLMs exhibit, and humans absolutely use when classifying. There are lots of awesome features such as the ability to inspect residuals and confidence intervals, and they’re generally computationally cheap. But for all that their absolute utility in the real world is fairly limited, especially when considering complex non linear tasks with huge latent spaces that are unobservable.
That means these baseline features are 100% fully negotiable. They're negotiable all the time because we employ humans that DON'T have these features. Thus LLMs don't need the features either.
Either way, we don't understand LLMs well enough to even predict whether future modifications will or will not have the features you claim. Such a hardline claim that it will never "fully be there" is illogical. Nobody predicted that transformers could lead to LLMs, nobody can predict what LLMs will lead to.