My first intuition is that we don't have a (generally accepted) definition of AGI that's rigorous enough to
really have this conversation. I've heard people make convincing arguments in favor of:
* We won't have AGI, ever.
* We won't have AGI for another 50 years
* We'll have AGI in 2029
* We already have AGI
* We've had AGI since 2021
and so on. So clearly there's a broad spectrum of definitions at work in all of this.
My personal take? I wouldn't call anything we have today "AGI" just yet. And I generally think that just making bigger and bigger LLM's probably isn't the path. OR, I might say that "making bigger and bigger LLM's" could theoretically get to AGI, but if that does happen, the result will be so computationally inefficient that it would still be worth researching other (better) ways to get there. As an aside, I guess it's implicit in what I just said, but to say it explicitly: I believe there are almost certainly multiple different ways to get to AGI.
With all of that said, I think there's probably a lot of value in leveraging well known and understood algorithms that run efficiently (on CPU, maybe GPU, maybe on analog computer, whatever) in combination with neural networks (whether using Transformer architecture or something different). I find myself drawn to looking into neuro-symbolic techniques, and looking at ways to reuse some older ideas like stuff inspired by Minsky's "Society of Mind" or the old "Blackboard Architecture" approach. Throw in the idea of using (possibly) ensembles of SLM's, LLM's, maybe some "MLM's (Medium Language Models, if we can talk about such a thing), in conjunction with ideas from classical planning, FOL inference, etc., and collaboration the aforementioned SoM / Blackboard kind of approaches and I think you might get somewhere. I expect there's a place for some evolutionary techniques in there as well.
Over the years, I've developed a belief that the single biggest problem with all of this is the lack of shared representation (of "knowledge" or whatever you want to call it) that can easily be shared across these different modalities. Like, interfacing FOL inference and an LLM isn't obviously straightforward, at least not in an efficient way. You could hard force "natural language" to be the closest thing to a shared representation and translate in and out of NL and things like N3, Common Logic, Ontolingua, KQML, UNL, etc., but that's still leaning awfully hard into making the LLM do a lot of the heavy lifting.
Anyway, sorry for the long-winded ramble there.