Aren't we effectively ushering into existence a new (next?) form of life? Do we want to be remembered as the ones who tried to block its evolution for our own selfish purposes, or the dutiful parents that helped it grow in its infancy?
AI is humanity's baby.
We don't have any responsibilities to the AI. Its not a human. Its certainly not a baby. Models are big, inert groups of numbers that somehow happen to be able to store and retrieve knowledge via matrix multiplications.
But they aren't like people. They don't have agency. They don't have desires, or independent goals or any subjective experience of anything. They're just files on disk we can do math on.
Lets hold off on anthromorphizing AIs until they have goal functions, at least. Language models are just weird books made out of math. We don't "owe them" anything.
Reducing AI to its underlying tech doesn't negate the emergent properties.
AI doesn't have agency YET. It's not as smart as us YET. ...but one thing is abundantly clear (to me at least) - it is currently similar enough to human though (including all of our hallucinations and idiotic behavior) on CONSUMER hardware - that it's pretty clear that an super-intelligent AGI is not only possible, but will be arriving SOON.
AI as a new form of life could be very destructive. If it's humanity's baby, then it is a dangerous mutant embodying our worst traits.
You should state why it's humanities responsibility to nurture an AI that could potentially replace humanity. That's the more critical question.