My main gripe about AGI is that everyone assumes a general intelligence will somehow be able to self optimize towards a more and more improved state never reaching a plateau. I think it's much more likely that the optimization landscape when searching for "higher intelligence" is full of local optima and does not have these "singularity" style ramps towards infinite intelligence that any self optimizing system could just discover and ride toward infinity.

There are millions(?) of human researchers (and orders of magnitude more computers) doing gradient free optimization through their research in this direction, and the progress is painfully slow, I know because I'm one of them. There are billions of years of optimization (evolution) towards this goal, and a total of (1) species has achieved any kind of notable intelligence. We are collectively giant parallelized optimization.

We already have "AGI" orders of magnitude more capable than any single human in the form of billions of people networked through the internet searching for fulfillment, money, power, fame, etc. for the next big discovery or supporting this effort by providing everything the entire global "machine" needs to run. The idea one little box running the right program can have access to the energy to beat this effort and exponentially improve things seems laughable in comparison.

The global "AGI" formed by all of us, the internet, and computers, is more likely to destroy society in the next 20 years in some catastrophic event than some paperclip machine.