Because ultimately people will be driven by the same thing that drove previous generations: money, power, fame, success, ego etc.
And no amount of university lectures with enlightened scientists and their superior values will change that.
Until that can be achieved, alternative options that generate more value will always be prioritised. Such is only the logical turn of events.
Does the primary responsibility for that really fall on engineers in the first place?
It just feels a little victim blame-y to me. Shouldn't most of the culpability fall on those stakeholders who don't want to be convinced because it means lower feature-velocity or less investor-profit?
I mean, do we blame food chemists for "failing to convince stakeholders" that the candy-company should use ethically sourced fair-trade chocolate and less sugar? Or mechanical engineers for "failing to convince stakeholders" that the oil-company should avoid drilling in areas that are environmentally-sensitive or controlled by warlords of questionable legitimacy?
No, we recognize that a laborer having specialized-skills doesn't automatically mean the corporate-machine is interested in their other opinions. (Unless they're visibly documenting problems that could someday harm the company in a potential lawsuit, in which case they tend to get laid off.)
b) CEOs typically will do whatever it takes to maximise revenue, past the point of what is ethical/moral, up until they hit the limit of whether it's illegal.
And then most but not all will stop. I'm hopeful but doubtful that AI will be any different.
"The paperclip maximiser was great up until it turned the solar system into paperclips.
Thankfully, a few of us were aware of the risk and escaped to a nearby solar system. It's not as nice as the first one, but it'll do.
Now, who wants to buy some paperclips? :)"
If picking the ethical choice maximized corporate profits, the Earth would have become a utopia a hundreds of years ago.
They generally don't care about the opinions of engineers that much, except to estimate the cost and practicality of different ideas.
And we've already let those AGI align themselves predominantly on "profit"
In their worry about a new AGI that they may never actually discover, the apocolypts are being stymied by the one that's already there and that's already working to take over the world.
There's a lot of conversation around this, one term used is "Egregore"
That's the definition being used in AI research. Does this system succeed on this test of {reasoning, theory of mind, abstract thinking, ... etc}? Can this system outperform most human workers?
And yes, corporations and other collectives can fully satisfy both criteria for most economically valuable tasks. If you make the same degree of input/output adaptations for them that you do for LLM's, they'll write you articulate letters all day, perform inhuman computations and reasoning tasks, develop and execute grand strategy, control all your robots, ace all your tests, and then write all your code for a tenth of what it would cost with your Bay Area tech bro.
From OpenAI's charter:
> OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity.
( This is the punny name that OpenAI employees use to refer to what's officially known as AI Existential Safety, explained here https://news.ycombinator.com/item?id=38374739#38375199 )
It's interesting I think because all of the above have ethics in my experience. What they don't do is treat their ethics with the same sense of supremism as the vocal activists/advocates who frame everything unaligned with their view critically. And if you show even a little compromise that too gets treated as a lack of ethics in that people will at the very least view it as the money winning out over personal beliefs.
The idea that someone can be legitimately ethical without being an extremist about the matter is largely unacknowledged by the vocal ethicists. But in a way that also makes sense because from their view the silence is probably just more room to criticize in the conversation.