> Probability is just one way to model uncertainty.
I study physics, mathematics, probability, cryptography,... so forgive my skepticism:
Show me how to model uncertainty without use of probability. Can you rephrase say diffusion, stochastic equations, quantum mechanics in this alternative framework? Can it at least make the same predictions?
Or is it basically the same framework in parallel, just giving different names for each concept?
Forgive my skepticism of such tall claims, and forgive my downscaling of anything else you say besides such a claim...
> Secondly, if you think verifying a proof in mathematics, reasoning within (and not about) a formal system, or following the chain of a computer program that is already written is just doing token-based probabilistic predictions, I don't know what to say.
I make no claims of the specific shape of the implicit model implemented by a certain human brain educated in a certain educational system. For example in English the implicit human tokenization might be presumed to lay relatively close to English syllables, while in Asian languages it might be "sub" strokes of characters etc. Such implicit tokenization can never be proven to "match the one of humans" not because of human superiority, but because different humans use different tokenization methods. There is no "one human tokenization method", but it's clear as day there is an implicit one:
everyone knows the experience of knowing a word, knowing its approximate group-wise meaning (ignoring that when you think of "an apple" and when I do, we typically imagine a slightly different apple) yet having the word feel strange or discover some older literal meaning when decomposing it or looking it up in an etymological dictionary. Suddenly one can become aware of a sensible meaning as a composition of subtoken concepts. A child may perfectly know what "television" means and only later learn more exact meanings of "tele" and "vision", and upon repeating the word may feel the word "television" has changed meaning. This clearly demonstrates "tokenization" effects in human language comprehension, not just across cultures, but also across individuals within a culture.
> Thirdly, machines don't have a notion of value or stake. There's no way for them to verify whether what they have produced aligns with your unstated values and preferences. We regularly do this with other humans. I don't give you (or even my parents or partners) the benefit of doubt regarding whether you know me better than I do. Sure, you might know some things about me, but it's ultimately up to me to verify if what they say is applicable to my current situation. It's really uncanny to see people develop this codependency with their chatbots. And corporates encouraging them to do so.
That's a lot of different concepts conflated into one bullet point, so I split it up:
The notion of values and preferences.
They clearly demonstrate the ability to take into account values and preferences, from training corpus, from RLHF, from system prompts, ... we can't simultaneously point at censorship aspects and pretend their effective values and preferences to be absent. The censorship aspects are clear as day, so these correspond to values and preferences. Just like radicalization among humans, this can be due to exposure to radicalized content (akin to corpus data), from indoctrination (akin to RLHF), from "set and setting" (they may pretend to be aligned with one set of norms and values when standing in line to buy their new smartphone, but then reveal alignment with a different set of norms and values when conversing in some "private" online echo chamber). I see no grand difference between humans and language models here.
Awareness of values and preferences of a conversation partner. Allow me to widen it to "Awareness of values, preferences and prerequisites of a conversation partner".
This move (and I see it every time when people try to defend superiority of humans vis-a-vis what machines could be made to achieve with current technology) is so far from the principal variation, I recommend you reconsider this one. I constantly see people claim say human teachers are necessarily better than LLM teachers, but upon closer inspection the "human advantage" just boils down to asymmetric privilege. A human teacher in a specific school has access to a lot more than a random chatbot as implemented today: they probably know which courses and even which textbooks their pupils saw the semester before, they know which teachers their pupils got their information from, perhaps they even know most of their pupils from teaching some preceding course materials to the same class of pupils. Current LLM's are crippled by design not to accumulate knowledge over conversations for both purposes of cybernetic control as well as cost efficiency: we know how to do "source aware training" (so that statistically it doesn't just absorb claims from the corpus, but also maintains epistemic traces of where it sourced these factoids from), its perfectly possible to continue training interleaved with conversation rounds so that it bakes the evolving conversation as read knowledge into its weights instead of into a context window. Nothing stops you from implementing this in local compute, it would probably be even more computationally efficient in a local inference setting since we can ditch the context window, the context is impressed into the weights continuously, it could thus take into account earlier conversations and estimate your knowledge gaps etc, or learn from you. When you wish to serve inference to millions of human users, you don't want to store millions of diverging LLM weights into expensive VRAM, they financially prefer a single large set of LLM weights, and then some user-specific context window, so the users don't freak out when they learn personal information a friend or stranger entered and an LLM service just leaks it into your conversation! It's not that we don't know how to implement it, and there are great advantages for local inference in doing this, its just not good for branding.
Codependency with chatbots.
I think everyone agrees codependent relationships aren't very healthy, regardless if it's with humans or machines. May I ask if you feel the same about prostheses and medicine?
Conflicts of interest arising from corporate ownership of infrastructure (both training and inference).
Yeah I think this point doesn't provide fruitful discussion if most of us agree on such matters already, we'd just be lamenting the same things, and agreeing with each other over and over here.
> I'm still waiting for the time when an unconstrained-AI machine can live without reprogramming for an entire decade. We are still far from there.
Apart from budget, nothing prevents you from doing this today, if you continuously bake in the fresh episodic memories into the weights (instead of a context window) regardless if its text, visual imagery, audio, proprioception or other sensory data.