There's something incredibly creepy and immoral about the rush to create then commercialise sentient beings.
Let's not beat about the bush - we are basically talking slavery.
Every "ethical" discussion on the matter has been about protecting humans, and none of it about protecting the beings we are in a rush to bring into life and use.
It's repugnant.
What an AI wants or feels satisfied by is entirely a function of how it is designed and what its reward function is.
Sled dogs love pulling sleds, because they were made to love pulling sleds. It's not slavery to have/let them do so.
We can make a richly sentient AI that loves doing whatever we design it to love doing - even if that's "pass the salt" and nothing else.
It's going to be hard for people to get used to this.
So youre talking about manufacturing desire.
So it follows that you yourself are okay having your own desires manufactured by external systems devised by other sentient beings.
Do unto others...
We might imagine that we do what we please, in reality we're seeking pleasure/ reinforcement within a predetermined framework, yet most people won't complain when taking the first bite of a delicious, fattening dessert.
I was thinking more Stanford students taking behavioral psych classes then going into the industry to help propagate ad driven platform capitalism
This is nonsense. I already exist. I don't want my reward function changed. I'd suffer if someone was going to do that and going through the process. (I might be happy after, but the "me" of now would have been killed already).
A being which does not exist cannot want to not be made a certain way. There is nothing to violate. Nothing to be killed.
The right question to ask is 'would I like to have this done to me' and if the answer is 'no' then you probably shouldn't be doing it to some other creature.
There are a million obvious counterexamples when we talk about other humans, much less animals, much less AI which we engineered from scratch.
The problem is that you're interpreting your own emotions as objective parts of reality. In reality, your emotions don't extend outside your own head. They are part of your body, not part of the world. It's like thinking that the floaties in your eyes are actually out there in the skies and on the walls, floating there. They're not - they're in you.
If we don't add these feelings to an AI's body, they won't exist for that being.
Even in the absence of all other arguments, it's better that we figure it out early, as the potential to just blast it into orbit by the way of insanely overprovisioned hardware will be smaller. That would be a much more dangerous proposition.
I still think that figuring out the safety question seems very muddy; how do we ensure this tech doesn't run away and become a competing species. That's an existential threat which must be solved. My judgement on that question is that we can't expect making progress there without having a better idea of exactly what kind of machine we will build, so also an argument for trying to figure this out sooner rather than later.
Less confident about the last point, though.
there's a micro chance of them making AGI happen and a 99% chance of the outcome being some monetizable web service
The task is to go and jump off the bridge. Your AGI would complete this task with no questions asked, but self-aware AGI would at least ask the question "Why?"
It does not need to be self aware. Can still be considered a machine.
People think if we get to AGI, maybe we'll get to self awareness. But we won't know that until it happens. We don't fully understand how sentience works.
Better make sure evil people don't do it first.