Not really. It's a derivative of that being thrown back at you via statistics.
No, there’s no evidence suggesting that.
If you are refering to the mathematical universe, its a fun mental excersise, not a serious idea.
It's actually the only idea that explains how the universe came to exist with nothing outside it and no creator.
It think that's the most boring and irrelevant question in human culture and I believe humanity is about a million years from tackling it in any competent manner. We have way more to find out about our smallness and irrelevance first.
I presume you are claiming by saying orthogonal that statistics are neither required nor preclude X.
You are using the presence of statistical processes as a claim against ChatGPT with
>It's a derivative of that being thrown back at you via statistics.
For this to have any weight given the orthogonal claim, you are asserting that LLMs are entirely statistics (the property you assert is orthogonal and thus carries no weight, yet invoking it as a property suggests that it is relevant, presumably because there are no other properties)
Then we get into the weeds of what you mean by statistics. It could be broad or narrow. If it is narrow then it is easy to claim neither sufficient nor precluding, but extremely hard to draw any other conclusion because any system can break free of the proof by including the tiniest bit that does not fit within the narrow definition of statistics. If the definition is broader to include everything that happens within ChatGPT, then you are faced with the question of what is not covered by that definition of statistics. How can you show that human minds do not operate within those limitations.
I don't think that ChatGPT is conscious in any sense that would be meaningful to a layman, but I also don't think that you can demonstrate it could not be purely based on the fact it uses any particular process without a rigorous proof of the limitations of that process.
The only way you could reasonably
The whole parrot idea is half baked. Yes, in closed book mode, and without a seed prompt that ads anything new to the existing knowledge of the model, it will parrot (adapt) from its training data. But if you add novel data in the prompt, then we push it outside its training distribution, and it learns on the fly. And usually all interactions bring some new elements to the model, new knowledge or skills, or new context. The more elaborate the prompts, the longer the session of messages, the less parroting takes place. The O3 model performance on ARC-AGI is another argument against the reductionist idea of parroting.
Or, it's not "statistics" as in they average the shit out and call it a day, like idk computing a census, or a report to your boss about the impact of most recent ad campaign, or something.
It's "statistics" in the sense of dealing with probability distributions, systematically teasing out correlations and causal relationships in inputs too fuzzy to handle symbolically. Of which language is literally the prime example.
All that talk about grammar and sentence structure and symbolic transformations and Chomsky etc. is just bullshit - it's not how we do it at all, which is obvious to anyone who was forced to learn grammar and sentence breakdown at school, and started wondering how come they already know to speak without knowing all this. Nah, we learn the language the same way LLMs do - by immersion and feedback, until we're able to say what "feels right" and be right. All those formalisms are basically fitting probability distributions with neat symbolic approximations, and are as valid as replacing Navier–Stokes equations with a linear function because that's how it looks like in typical case and we don't know any better.
So yeah, it's all "thrown back at you via statistics". So is all of the human thought it's derivative of. Embrace it: we're all "statistical machines".
And you know this how?
> we learn the language the same way LLMs do
Based on?
> So yeah, it's all "thrown back at you via statistics". So is all of the human thought it's derivative of
You say this based on what? Do you have any real research to support any of this? Because from what I see in the neuroscience community, LLMs are not even remotely an approximation of a real brain, despite being built of "neurons" (which are nothing like real neurons).
Probability theory, computability theory. There's no other way.
> Based on?
High-level description of the learning approach, and approximately the entirety of recorded human history?
> Because from what I see in the neuroscience community, LLMs are not even remotely an approximation of a real brain, despite being built of "neurons" (which are nothing like real neurons).
So what? Neurons in silica are to neurons in vivo as my Twitter clone in 15 lines of terse Perl is to the actual web-scale monstrosity a Twitter or other modern scalable SaaS is. I.e. it does the same core thing faster and better, but obviously lacks incidental, supporting complexity to work in the same environment as the real thing.
Yes, a biological neuron is huge and complex. But a biological neuron is, first and foremost, an incrementally evolved, independent, self-replicating and self-maintaining nanomachine, tuned to be a part of a specialized self-assembling system, filling a role in a large and diverse ensemble of other specialized nanomachines. Incidentally, a neuron also computes - but it's not computing that uses this complexity, it's all the self-replicating self-assembling nanomachine business that needs it.
There's absolutely no reason to believe that neurons in silica need to be comparably complex to crack intelligence - not when we're free to strip the problem out of all the Kubernetes/protein nanomachine incidental complexity business, and focus on the core computation.
> Probability theory, computability theory. There's no other way.
There is no credible evidence that humans learn via statistics and significant evidence against (poverty of stimulus, all languages have hierarchical structure). One other way was proposed by Chomsky, which is that you have built-in machinery to do language (which is probably intimately related to human intelligence) just like a foal doesn't learn to walk via "statistics" in any meaningful sense.
> Neurons in silica are to neurons in vivo ...
Again not true. Observations about things like inter-neuron communication time suggests that computation is being done within neurons which undermines the connectionist approach.
You've just presented a bunch of your own intuitions about things which people who actually studied the field have falsified.
Except living among humans on planet Earth, that is.
Even the idea of a fixed language is an illusion, of the same kind like idk "natural balance", or "solid surface", or discrete objects. There's no platonic ideal of Polish or English that's the one language English speakers speak, that they speak more or less perfectly. No singular the English language that's fundamentally distinct from the German and the French. Claiming otherwise, or claiming built-in symbolic machinery for learning this thing, is confusing map for territory in a bad way - in the very way that gained academia a reputation of being completely out of touch with actual reality[0].
And the actual reality is, "language" is a purely statistical phenomenon, an aggregate of people trying to communicate with each other, individually adjusting themselves to meet the (perceived) expectations of others. At the scale of (population, years), it looks stable. At the scale of (population, decades), we can easily see those "stable" languages are in fact constantly changing, and blending with each other. At the scale of (individuals, days), everyone has their own language, slightly different from everyone else's.
> poverty of stimulus
More like poverty of imagination, dearth of idle minutes during the day, in which to ponder this. Well, to be fair, we kind of didn't have any good intuition or framework to think about this until information theory came along, and then computers became ubiquitous.
So let me put this clear: a human brain is ingesting a continuous time stream of multisensory data 24/7 from the day they're born (and likely even earlier). That stream never stops, it's rich in information, and all that information is highly coherent and correlated with the physical reality. There's no poverty of language-related stimulus unless you literally throw a baby to the wolves, or have it grow up in a sensory deprivation chamber.
> all languages have hierarchical structure
Perhaps because hierarchy is a fundamental concept in itself.
> you have built-in machinery to do language (which is probably intimately related to human intelligence)
Another way of looking at it is: it co-evolved with language, i.e. languages reflect what's easiest for our brain to pick up on. Like with everything natural selection comes up with, it's a mix of fundamental mathematics of reality combined with jitter of the dried-up shit that stuck on the wall after being thrown at it by evolution.
From that perspective, "built-in machinery" is an absolutely trivial observation - our languages look like whatever happened to work best with whatever idiosyncrasies our brains have. That is, whatever our statistical learning machinery managed to pick up on best.
> like a foal doesn't learn to walk via "statistics" in any meaningful sense.
How do they learn it then? Calculus? :). Developing closed-form analytical solutions for walking on arbitrary terrain?
> Observations about things like inter-neuron communication time suggests that computation is being done within neurons
So what? There's all kinds of "computation within CPUs" too. Cache management, hardware interrupts, etc., which don't change the results of the computation we're interested in, but might make it faster or more robust.
> which undermines the connectionist approach.
Wikipedia on connectionism: "The central connectionist principle is that mental phenomena can be described by interconnected networks of simple and often uniform units."
Sure, whatever. It doesn't matter. Connectionist models are fine, because it's been mathematically proven that you can compute[1] any function with a finite network. We like them not because they're philosophically special, but because they're a simple and uniform computational structure - i.e. it's cheap to do in hardware. Even if you need a million artificial neurons to substitute for a single biological one, that's still a win, because making faster GPUs and ASICs is what we're good at; comprehending and replicating complex molecular nanotech, not so much.
Computation is computation. Substrate doesn't matter, and methods don't matter - there are many ways of computing the same thing. Like, natural integration is done by just accumulating shit over time[2], but we find it easier to do it digitally by ADC-ing inputs into funny patterns of discrete bits, then flipping them back and forth according to some arcane rules, and then eventually DAC-ing some result back. Put enough bits into the ADC - digital math - DAC pipeline, and you get the same[1] result back anyway.
--
[0] - I mean, what's next. Are you going to claim Earth is a sphere of a specific size and mass, orbiting the Sun in specific time, on an elliptical orbit of fixed parameters? Do you expect space probes will eventually discover the magical rails that hold Earth to its perfectly elliptical orbit? Of course you're not (I hope) - surely you're perfectly aware that Earth gains and loses mass, perfect elliptical orbits are neither perfect nor elliptical, etc. - all that is just approximations safe at the timescales usually consider.
[1] - Approximate to an arbitrary high degree. Which is all we can hope for in a physical reality anyways.
[2] - Or hey, did you know that one of the best ways to multiply two large numbers is to... do a Fourier transform on their digits and summing them up in the frequency domain?
(Incidentally, being analog, nature really likes working in the frequency domain; between that, accumulation as addition, and not being designed to be driven by a shared clock signal it should be obvious that natural computers ain't gonna look like ours, and conversely, we don't need to replicate natural processes exactly to get the same results.)