Until that can be achieved, alternative options that generate more value will always be prioritised. Such is only the logical turn of events.
And we've already let those AGI align themselves predominantly on "profit"
In their worry about a new AGI that they may never actually discover, the apocolypts are being stymied by the one that's already there and that's already working to take over the world.
There's a lot of conversation around this, one term used is "Egregore"
That's the definition being used in AI research. Does this system succeed on this test of {reasoning, theory of mind, abstract thinking, ... etc}? Can this system outperform most human workers?
And yes, corporations and other collectives can fully satisfy both criteria for most economically valuable tasks. If you make the same degree of input/output adaptations for them that you do for LLM's, they'll write you articulate letters all day, perform inhuman computations and reasoning tasks, develop and execute grand strategy, control all your robots, ace all your tests, and then write all your code for a tenth of what it would cost with your Bay Area tech bro.
From OpenAI's charter:
> OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity.
Does the primary responsibility for that really fall on engineers in the first place?
It just feels a little victim blame-y to me. Shouldn't most of the culpability fall on those stakeholders who don't want to be convinced because it means lower feature-velocity or less investor-profit?
I mean, do we blame food chemists for "failing to convince stakeholders" that the candy-company should use ethically sourced fair-trade chocolate and less sugar? Or mechanical engineers for "failing to convince stakeholders" that the oil-company should avoid drilling in areas that are environmentally-sensitive or controlled by warlords of questionable legitimacy?
No, we recognize that a laborer having specialized-skills doesn't automatically mean the corporate-machine is interested in their other opinions. (Unless they're visibly documenting problems that could someday harm the company in a potential lawsuit, in which case they tend to get laid off.)
If picking the ethical choice maximized corporate profits, the Earth would have become a utopia a hundreds of years ago.
"The paperclip maximiser was great up until it turned the solar system into paperclips.
Thankfully, a few of us were aware of the risk and escaped to a nearby solar system. It's not as nice as the first one, but it'll do.
Now, who wants to buy some paperclips? :)"
( This is the punny name that OpenAI employees use to refer to what's officially known as AI Existential Safety, explained here https://news.ycombinator.com/item?id=38374739#38375199 )
They generally don't care about the opinions of engineers that much, except to estimate the cost and practicality of different ideas.
b) CEOs typically will do whatever it takes to maximise revenue, past the point of what is ethical/moral, up until they hit the limit of whether it's illegal.
And then most but not all will stop. I'm hopeful but doubtful that AI will be any different.