I don't think neural networks, language models, machine learning etc.. are even close to a general intelligence. Maybe there is some way to combine the two. I have seen some demonstrations of very primitive clusters of brain cells being connected to a computer and used to control a small machines direction.
If there is going to be an AGI I would predict this is how it will happen. While this would be very spectacular and impressive I'm still not worried about it because it would require existing in the physical world and not just some software that can run on any conventional computer.
I'm of the mind is there is likely many was of simulating/emulating/creating intelligence. It would be highly surprising if there was only one way, and the universe happened to achieve this by the random walk of evolution. The only question for me is how much work is required to discover these other methods.
My thought process on why it might not be possible in principle with conventional computer hardware is how perfect its computations are. I could be completely wrong here, but if you can with perfect accuracy fast forward and rewind the state of the simulation then is it actually intelligent? With enough time you could reduce the whole thing to a computable problem.
Then again maybe you could do the same thing with a human mind. This seems like a kind of pointless philosophical perspective in my opinion until there is some way to test things like this.
I would love to know one way or the other on the feasibility of AGI on a silicon CPU. Maybe the results would determine that the human mind is actually as pre-determinable as a CPU and there is no such thing as genral intelligence at all.
I don't see how the conclusion follows from the premise.