What people like Sam Altman claim the end goal is? ASI (Artificial Super Intelligence) ~~takes over~~ optimizes the world, abolishing suffering and death forever. Well, hopefully it's not too nasty, living in someone else's dream and all.
We all have tiny bits of agency / freedom but for the most part our lives are bounded and constrained by the gigantic complex systems which support yet trap us.
Ain't no way the millionaires and billionaires of the world are going to accept being on equal footing with the poors, haha. Outside of the folks that fear Skynet, I imagine this is the real concern with alignment.
How can a being of pure logic look past the hoarding of resources? If a monkey in captivity was hoarding all the food we'd step in and make sure the other monkeys had food to eat. I have to imagine the next intelligence above us would consider doing the same for us if given the power to do so.
We'll never get to a point without an "Us vs Them", or "Have and Have Nots" - it's tragic, and I really hope I'm wrong, but I have infinite doubt on this one.
Having said that I'll give anything a go, we're still stuck in the age of monarchies over here on Normal Island. AI wants to take over? I'm down to give it a shot as long as it promises to not wear a crown during a recession, haha
If I could accurately predict what a superintelligence's goals would be I'd have a decent career advising AI companies on their alignment concerns :)
Just taking a swing at it with this ol' legacy wet brain because it's fun to think about. There's nothing to buy, it is an opinion. Heck, it's practically just an exercise in writing science fiction at this stage.
What do you think it would do?