I see you pushing back on related topics in a few comment threads. I’m not sure if I can define “understanding” but I think “intelligence” is a fair synonym for this conversation. I see your passion for clarifying that LLMs are not intelligent, and I’m sensing you’re conflating that with “consciousness” and trying to hedge against other people inadvertently making that same error.
Intelligence is not uniquely human. And not constrained to consciousness. Even consciousness itself, under fringe definitions like exhibiting non-deterministic behavior, is not constrained to living beings. Systems, for example, exhibit intelligence all the time - groups of people, colonies of ants, weather systems, self-organized criticality like blooming forest fires or abelian sand piles, evolutionary processes, and so on.
Your counter argument - that LLM responses are “just” statically probable (i.e. predictions) - actually supports this notion. Many intelligent statements we make are probably right or probably wrong, in part or in whole. We as humans are far more than statistical computers, but our intelligence in isolation doesn’t seem far off from that of LLMs (not mechanistically, but rather qualitatively).
3Blue1Brown recently posted a video called something like “Compression is Intelligence”. Worth a watch.
The problem with all of this is that because an LLM gives a facsimile of language and conversation people believe it can "comprehend" or "understand" or whatever word you want to use. This problem isn't new, AI research has had this issue for decades. The risk is that this makes people misunderstand and offload their decision making to it, treating it like a person to be instructed or reasoned with. It isn't, and can never be.