And if anyone believes the AGI hyperbole, oh boy I have a bridge and a mountain to sell.
LLM tech will never lead to AGI. You need a tech that mimics synapses. It doesn’t exist.
LLMs have their name for a reason: they model human language (output given an input) from human text (and other artifacts).
And now the idea seems to be that when we do more of it, or make it even larger, it will stop to be a model of human language generation? Or that human language generation is all there is to AGI?
I wish someone could explain the claim to me...
And because there's something in the human mind that has a very strong reaction to being talked to, and because LLMs are specifically good at mimicking plausible human speech patterns, chatGPT really, really hooked a lot of people (including said VC/private money people).
It's not that language generation is all there is to AGI, but that to sufficiently model text that is about the wide range of human experiences, we need to model those experiences. LLMs model the world to varying degrees, and perhaps in the limit of unbounded training data, they can model the human's perspective in it as well.
Why would you think synapses (or their dynamics) are required for AGI rather than being incidental owing to the constraints of biology?
(This discussion never goes anywhere productive but I can't help myself from asking)
LLM are really good at pretending to be intelligent but I don't think they'll ever overcome the "pretend" part.
I suspect this may be one of those predictions that may not quite pan out. I am not saying it is a given, but never is about as unlikely.
Like at the moment I think during training new data changes all the model weights which is very compute intensive and makes it hard to learn new things after training. The human brain seems to do it in a more compartmentalised way - learning about a new animal say does not rewrite the neurons for playing chess or speaking French for example. You could maybe modify the LLM algo along those lines without throwing it away entirely.
So you might improve algorithms (by doing matrix multiplications in a different order.... it's always matrix multiplications) but you'll be feeding them junk.
So they need ever increasing amounts of data but they are also the cause of the ever increasing shortage of good data. They have dug their own grave.
'Machine will always know where to go from here on now'.
I think _maybe_ quantum computing might be the tech that moves AGI closer. But I'm 99.9999% certain it won't be LLM tech. (Even I can't seriously say 100% for most things, though I am 100% certain a monkey will not fly out of my butt today)
It’s not the hardware, it’s the algorithms.