The assumption that the trajectory of LLMs leads to an actual AGI is, frankly, ludicrously optimistic. The main thing these things
should be demonstrating to us is that language learning, alone, is neither equivalent to, nor a reliable signal of, general intelligence. A myna bird with orders of magnitude more long term memory does not a destroyer of humanity make,
unless a human with something considerably less than general intelligence decides to hand control of a nuclear arsenal to that bird.
That said, the arrival of an actual AGI on the scene would be roughly the same as the arrival of another actual NGI… we most likely wouldn’t recognize it, wouldn’t know how to communicate with it, and would both misunderstand it and attempt to enslave and exploit it to our primarily economic ends, if we didn’t kill it reflexively because we are considerably more governed by our emotions than we are by what we presume is our intellect. And that’s assuming an LLM trained to and beyond some magical cognition emergence event horizon with the sum total of human internet-accessible verbiage didn’t kill itself instantaneously, which would be the most likely outcome of waking up to human-level general intellect with neither a body or the emotions required to understand the vast majority of the human-level “knowledge” such a model possessed.
In other words I don’t think it would do much to human society other than prove that humans aren’t as smart as they think they are.