I've been thinking lately about how AGI runs up against the No Free Lunch Theorem. This is what irritates me: science is not determining the narrative. Money is. I highly recommend mathematician David Wolpert's work on the topic. I think he inadvertently proved that ASI is physically impossible. Certainly he proved that AOI (artificial omniscient intelligence) is impossible.

One thing he showed is that you can't have a universe with two omniscient intelligences (as it would be intractable for them to predict the other's behavior.)

It's also very questionable whether "humanlike" intelligence is truly general in the first place. I think cognitive neurobiologists would agree that we have a specific "cognitive niche", and while this symbolic niche seems sufficiently general for a lot of problems, there are animals that make us look stupid in other respects. This whole idea that there is some secret sauce special algorithm for universal intelligence is extremely suspect. We flatter ourselves and have committed to a fundamental anthropomorphic fallacy that seems almost cartoonishly elementary for all the money behind it.