While doesn't matter much in the rest of the policy, this is a common misconception among AI skeptics. It is not the case for a long time (since RL is used heavily in the training) and a LLM may go beyond its training data.
Saying a LLM is all just based on probabilities is pretty meaningless, even if it is true, since everything else is just based on probabilities too. What matters is the massive amount of machinery that generates those probabilities. Anything from rolling a dice to an accurate model of an entire human, or even a model of the entire universe, is all "just probabilities".
But there are more probabilistic elements about LLMs. Inference is by default deterministic for any given input and weights, however it is made stochastic algorithmically, when the model chooses the second or third likeliest outcome with some probability. Training is stochastic, the adjustment of the weights is probabilistic, etc. etc.
You seem to make the fact that our brains are also probabilistic as some sort of a gotcha. However that has never been in dispute. Neurons fire with certain probability, this has been known since the advent of neuroscience. However this is why computers are preferred for certain tasks over human brains. When you run a program you can be certain it behaves in a certain way given a particular input and parameters.
I don‘t think I have ever seen anybody make a claim that LLMs are a bad (or otherwise limited) technology because of the probabilistic nature of it. Plenty of excellent algorithms (including other machine learning algorithms) are probabilistic and work excellently for what they are meant to do. Problems arise when the algorithm is used for something more, and that is the case against LLMs. It is a next token prediction algorithm that people are using to write software. If they treat it as a compiler it will be lacking, and LLMs being non-deterministic is one of many reasons for why LLMs are bad compilers (or more accurately; are in fact not compilers).
And for that matter, neither are humans compilers. If you find an AI-hater who says “humans are good compilers” then I will agree with you that that person is wrong.
However the loss function (or the success criteria more broadly) for the game of go is extremely simple. We do not know whether such a success criteria even exists for a generic task like coding, and it is a mistake to assume that the AI labs have found one.
Like you said, you can have reinforcement learning which doesn’t use training data. But that is not what my parent said. What they said is: since RL is used heavily in the training. And since reinforcement learning is a broad category which includes supervised learning, nothing in their logic disproves the strawman they created from an AI skeptic.
The details matter (training set, full context). I doubt our current LLMs knows what smelling cut grass on a wet morning feels like, or how the stomach flutters on a first kiss - even if they have "read" tens to hundreds of attempts at describing such things.
You're essentially suggesting we play Turing's Guessing Game; if your LLM can guess my response to any prompt - then it can be said to have modeled how I would reply to those prompts. Expand the prompts far enough, and you could reasonably argue the LLM models me.
But you said:
>Training shifts the probability - but doesn't change the fact that the output is a sampling based on input and the model?
So which is it?
>You're essentially suggesting we play Turing's Guessing Game; if your LLM can guess my response to any prompt - then it can be said to have modeled how I would reply to those prompts. Expand the prompts far enough, and you could reasonably argue the LLM models me.
I'm merely noting that your argument that "training+input=>output is the problem" applies to humans. There's other arguments for and against LLMs however I'm simply pointing out that your argument isn't a good one.
I'm not sure I ever said that it was a problem.
I also never meant to say that sampling across the model (formed by training) based on input couldn't produce novel token sequences.
As for human vs LLM - my argument would be that while trained on a large corpus, the models are poor in experience and "sensory input" - their training so different from growing up - that even if we could model humans as LLMs - the gap between current LLMs and "human" LLMs remain a wide one.
More to the point - I believe the current generation of LLMs are way too sensitive to input (prompt, system prompt, context/RAG) - precisely because they're still too narrowly trained, and also aligned to weigh input heavily.
What we have today are models that work well with language (generate, transform, translate) - not so great on complex tasks - because they're still generating next probable token.