Something something LLMs can only predict the next word.
I hate to spin up this trendy debate again, but it's always funny to me to see the dissonance when talking about the exact same things in biological and mathematical cases.
The human mind is _so much more_ than a prediction machine, and incredibly complex... All that's before you get into the way the endocrine system interacts with your mind.
A single neuron has an average of 250000 connections in some parts of the brain. The speed at which neuronal signals travel varies neuron to neuron from 2.5m/s to 200m/s.
Human minds are more than just prediction. The anterior lateral prefrontal cortex has the sole responsibility of prediction (not that nothing else does, just that the ALPC seems solely dedicated to that task) and is extremely good at it. Prediction can influence all sorts of mental processes such as most all forms of perception... But it is _not_ the same as _being_ all forms of perception. If something unpredictable enough happens in front of you: you'll still see it.
Sure there are limits to that: when focused on a task the predictive parts of sight tend to filter out visual data that doesn't match the signal you're looking for (see: basketball players passing the ball and a moon walking man in an ape suit) but if every basketball player turned into Spaghetti-os and started screaming you'd still hear the sounds and see the O's.
So sure: LLMs do a good job at basic prediction but they're nowhere near the complexity of the human mind, of which prediction is only a small piece.
(And let's not even get into efficiency... A brain runs on 20W of power)
A bunch of cells likely have more than a few GB of data between them, but most of it is just instructions on how to fold proteins to replicate themselves. Is that intelligent behavior or just mindlessly executing a script, albeit in DNA form? I would lean towards the latter.
I suppose it's hard to compare biological intelligent systems directly since they all have to process all available sensory info in a timely manner, not just very narrow text input. Once there are more LMMs trained on vision, sound, and kinematic data it should get far easier to draw comparisons.
> but if every basketball player turned into Spaghetti-os and started screaming you'd still hear the sounds and see the O's
Unless of course someone had set up a Somebody else's problem field generator around them :)
At the end of the day this is mostly meaningless.
Biological creatures have to be efficient because they need to harvest energy from the environment around them. But to limit machines to that would be like saying we have to feed biomass to airplanes.
There’s hardly a comparison. If they want to, I’d like to see them start with using dynamic chunking instead of tokenization like the mind does. Also, they can tell me what pre-training method uses 20W per second for 1+ trillion parameters. I could use that right now.
This the right term to use here.
> Something something
If human brains have a model, then is language the transport layer on top of that? Is trying to get to intelligence via language no better than trying to get to "google" by modeling its TCP/IP traffic?
It's especially worse when low-quality popular science journalism promotes this notion, like this Quanta article about the human vision system working just like transformers do.
How do you know what job moffkalast has, and why does it matter? This reads like pure snobbery to me.
(Also: moffkalast did not in fact suggest that anything is a solution to a centuries-old problem. "Some common rhetoric about LLMs is too simplistic" is a far cry from "LLMs resolve all the perplexities about human consciousness and thought".)