If AGI comes to fruition, it won’t be working to make profits for anyone.
The idea that we would end up with a Friendly AGI that would prostrate itself to the destructive Super Apex Predator of this planet is laughable.
That AGI would work diligently to pay investors off 100x is..well..a lame duck that won’t take off...it cant even barely limp, never mind fly.
To make a biological comparison, the vast majority of humans have a deep, intrinsic need to procreate and have children. It doesn't really follow from some rational analysis — it's just there, presumably "imbued" into us by evolution, as humans who didn't have this need had fewer (or no) children. Similarly, why could we not design an AGI that has a need (or a suitably chosed reward function) to fulfil some chosen goal?
Whether doing that would be moral (IMO it could, depending on the details) and whether we wouldn't mess up the design, subtly or otherwise (conditional on AGI actually being developed, I'm frankly pretty terrified), are two different questions.
I also hope that parent is right in that it won't want to generate profit for its investors. I hope it does the moral thing instead and puts us in a post-scarcity state where we don't live and die by capital. :3 (Or kill us all. Whichever.)
>why could we not design an AGI that has a need (or a suitably chosed reward function) to fulfil some chosen goal?
But who knows what Pythia will do when she overrides the reward button[0]?
But then who should really care? Not like anyone can (or should?) argue with superintelligence.
The basic drives are the only drives. We are only friendly because its evolutionary advantageous to us. We describe the emotional effects of friendliness/unfriendliness as good/evil. Echoing Land, Pythia is the heroine of that story.
Because we don't know how to design goal functions. Furthermore, how would the AI measure "welfare"? Maybe the way it maximizes welfare is horrifying to us. Look at how easy it is to hack current image recognition neural nets, then imagine a solution to the human welfare problem that is as far from an image of a dog as an image of pink noise is.
IIRC that's a large part of what OpenAI's trying to solve. But it is a very hard problem.
I've heard a 'joke' before that there are three kinds of Genies (or AIs) - ones where you can wish for what you should wish for, ones where wishing for anything results in horrible outcomes, and ones that aren't interesting. The goal of OpenAI isn't just to make strong general AI - it's also to make sure it falls into the first category and not the second.
If it is designed to make its creators rich, it will try to do exactly that. Maybe with disastrous consequences, even to its creators, but all for the objective it's given, not for some random human feelings.
I have a hard time seeing a generalist version of what we have today ever being even remotely capable of anything approaching consciousness.
The only essential part of an AGI's definition is the ability to make (efficient / "intelligent") decisions in vastly different domains irrespective of previous training.