Can we bootstrap our way to basic AGI by tinkering with neural network architectures and scaling up the hardware? Maybe. After all, the human brain sort of echos that approach. But at some point, in order to sustain continued improvement toward optimal AGI, there will have be a strong algorithmic component to sequence prediction that is based upon a very deep understanding of what is computable (assuming the physical Church-Turing thesis). I don’t really see a way around that, because the foundational principles of algorithmic and computational complexity ultimately determine the upper limit of our ability to predict the future, which is kind of mind-blowing to me (and even more so considering that much of the theory was developed over half a century ago).
But wait! What about the halting problem, NP hardness, the NFL theorem, Gödel’s incompleteness theorems, Blum’s speedup theorem, ..., [insert your favorite pessimistic no-go theorem]? Yeah, so what? Most of these nonstarters apply to “almost all” valid problems, which ironically happen to overlap with “almost none” of the problems we care about, because the distribution of real world problems does not coincide with the distribution of problems randomly sampled from a formal language. If that were the case, then nothing would be predictable at all because prediction-making beings could not exist in such an environment (in other words, real world problems tend to exhibit Kolmogorov-compressibility in the form of mathematical substructure that leads to heuristic solvers that are particularly effective beyond what average-case complexity would imply).