I would've hoped he'd be exploring weirder alternatives off the beaten path. I mean, neural networks might not even be necessary for AGI, but no one at OpenAI is going to tell Carmack that.
Otherwise you may end up walking the ditch beside the beaten path. It is slow and difficult, but it won't get you anywhere new.
For example, you may try an approach that doesn't look like deep learning, but after a lot of work, realize that you actually reinvented deep learning, poorly. We call these things neurons, transformers, backpropagation, etc... but in the end, it is just maths. If you end up finding that your "alternative" ends up being very well suited to linear algebra and gradient descent, once you have found the right formulas, you may realize that they are equivalent to the ones used in traditional "deep learning" algorithms. It help to recognize this early and take advantage of all the work done before you.
I mean, any idiot can go off-trail and start blundering around in the weeds, and ultimately wind up tripping, falling, hitting their head on a rock, and drowning to death in a ditch. But actually finding a new, better, more efficient path probably involves at least some understanding of the status quo.
Oh man, you had me going with such a vivid metaphor. I was really hoping for a payoff in the end, but you abandoned it. The easy close would be "probably involves at least some understanding of the existing terrain" but I was optimistic for something less prosaic.
It would have been interesting seeing someone like Carmack going in this direction, but from the little details he gave he seems less interested in cells and Kjeldahl flasks and more of the same type-a-type-a on the ol' QWERTY.
† 'simply' might involve multiple decades of research and Buffett knows how many billions
[1] Human neurons implanted in mice influence behavior, https://www.nature.com/articles/s41586-022-05277-w
> Their survival here is incredible enough; but even more fantastic, to me, is the fact that they have gone unnoticed during this century, until now. Lately there have been men capable of appreciating their potential value– and not only myself. What Thon Kaschler might have done with them while he was alive!– even seventy years ago."
> The sea of monks' faces was alight with smiles upon hearing so favorable a reaction to the Memorabilia from one so gifted as the thon. Paulo wondered why they failed to sense the faint undercurrent of resentment– or was it suspicion?– in the speaker's tone. "Had I known of these sources ten years ago," he was saying, "much of my work in optics would have been unnecessary." Ahha! thought the abbot, so that's it. Or at least part of it. He's finding out that some of his discoveries are only rediscoveries, and it leaves a bitter taste. But surely he must know that never during his lifetime can he be more than a recoverer of lost works; however brilliant, he can only do what others before him had done. And so it would be, inevitably, until the world became as highly developed as it had been before the Flame Deluge.
-- A Canticle for Leibowitz
Neuralink is the only place where this pattern seemed to break a bit but then seems like Elon came into his own path with trying to push for faster results and breaking basic ethics.
This didn't happen
The denial of obviously fertile paradigm feels like such a useless self-defeating loss to indulge in an intellectual status game.
We could be all better off right now if connectionists were given DOE-grade supercomputers in the 90s, and were supplied with custom TPUs later in the 00s as their ideas were proven generally correct via rigorous experimentation on said DOE supercomputers. This didn't happen due to what amounts to academic bullying culture: https://en.wikipedia.org/wiki/Perceptrons_(book)
The sheer scale of cumulative losses we suffered (at least in part) due to this denial of the connectionism as a generally useful foundational field will be estimated somewhere in the astronomical powers of ten in the future, where the fruits of this technology will provide radically better lives for us and our descendants.
I see you have a knee-jerk reaction to hype and industry, and we are all fearing replacement unless its a stock market doing the work for us ... but why do you feel the need to punch down at this prosaic field "about nonlinear optimization"? The networks in question just want to learn, and to help us, if we train them to this end - and we make any and all excuses to avoid receiving this help, as our civilization quietly drowns in its own incompetency...
Of course, there are a group of people defending the symbolic computation, e.g. see Gary Marcus, and always pushing back on connectionism (neural networks).
But this is somewhat a spectrum, or also rather sloppy terminology. Once you go away from symbolic computation, many things can be interpret as neural network. And there is also all the computational neuroscience, which also work with some variants of neural networks.
And there is the human brain, which demonstrates, that a neural network is capable of doing AGI. So why would you not want a neural network? But that does not say that you can do many things very different from mainstream.