Nothing is keeping this from becoming AGI (for some definition of AGI), and you should not be panicking, remember that nobody can predict the future, especially those who claim they can, and that includes yourself.
And "nothing is keeping this from becoming" is quite a confident prediction of the future, which you said nobody could make. Well, maybe not, but how can you be so sure? There might be fundamental issues with current ML that you just can't scale away.
Remember mechanical computers? They were improved upon for decades, but there was no chance at all they would ever reach the completeness and computing power of modern electronic computers. A new paradigm was needed for that, transistors had to be invented.
The same might apply to current approach to "AI" through ML. Maybe, maybe not. So let me pull an old quote for all of you who claim current ML can evolve into AGI: Show me the code. Show me the program capable of running AGI on hardware which is basically still just flipping transistors. Then I believe you.
There is nothing scarier than to think that next word predictor can be sentient, because it means we are just next word predictors, and all our pain and suffering is beyond meaningless.
In the same time I think there are definitions for AGI that can fuel exponential development in all important fields (from protein folding to medicine to enegry and etc), and this can just as well be a transformer large enough that has reasoning.
Btw I an not scared, just unconvinced. And you arguing like a high priest with all that talk about pain and suffering ain't going to change that.