But since we can't yet define consciousness, how do you know the AGI is just 'faking it' instead of being the real deal? What are human brains but a bunch of neurons pretending to be a person?
I suspect that a lot of the complexity is legacy code and/or incidental to the implementation. Cells have to do a lot of things just to continue living. Likewise, a lot of what the brain does is just control the very complex biochemical reactor that it lives in. ie: whales have big brains relative to humans (though they are a much smaller % of their body).