Ultimately it's a tool, until I suppose it becomes fully self directed, maybe.
Someone, sometime, somewhere wrote an API to aide in care of puppies. Said Algo is also fully capable of determining the best course of applying the genocide of puppies, provided key inputs are flipped to negative.
I have a voice activated box that plays music, sometimes the wrong one. I don't scream at it unlike my family. It doesn't understand emotional inflection, tone, etc. It may be program to understand that at some point. and realize the operators angry. Even at that point, I would not ever consider it human or anything other than a neutral device.
Take it even further, to weapons, devices entirely designed to destroy a human life, is still a tool and neutral as the tool is unaware of the intended use, whether it is for offense or defense, the taking or preservation of life through its intended use.
When these models are trained, I am not a big fan of them being corralled or limited. The body of work that the model is trained on and the weights that allow for the adjustment should allow for accurate representations of the computations and calculations.
I have children and I do not lie to them. I also speak to them in a way that is age appropriate. How is this different? For starters, I am not a child, and when I am interacting with a system, not a parental or overseer, I do not want to be treated as a child. I can handle the world and all of its uglyness.
I do not mind the hallucinations, inaccuracies, and other mistakes driving from computations.
Example: I recently created a prompt for a history of adoption, property transfer, including parental rights history, and was lectured about slavery. How can I trust the results if the prompt is being interpreted and the answer modified dependent upon an unrelated subject that someone finds sensitive.