In essence the AI is just giving us what the majority of it's test data tells it that we want to hear.
But the whole point is, there's no such thing as "objective", especially when you start talking about people. There's no place to stand as a human being that doesn't involve experiencing the world through one's own consciousness, which has an implicit value structure. Everyone has their own little "religion", and when they share that worldview with other people, the religion then becomes that which holds the group together. "Ligament" (that which holds bones together) and "religion" (that which holds groups together) have the same root: "lig" which means "to bind".
AI is revealing this to a large segment of our population that considers themselves "objective" and "secular", and the cognitive dissonance that is coming out as a result is funny and disturbing in equal parts.
I agree with you.
However I would add to your statement about AI being revealing. One thing that AI I feel is revealing to me at least, is that it's now making me think much more about the distinction "a human search for truth" however flawed that may be, and things that are made simply to appeal to my biases.
Linking this back to the topic at hand, maybe there is no such thing as objective, but there is a difference between a rule written by a human that was at least trying to search for truth VS a rule written by an system that just parrots the most pleasing thing back at us.
This is essentially the same situation as people who have never left their state, or their home country. Many things they take for granted and that they consider to be "basic human nature" actually end up being very cultural and very uncommon once one starts traveling outside one's home country. But you are completely unaware of these situations because they never get challenged when you live in a culture that instantiated itself in you as you grew up.
You can see how truth starts to shake right ? It is an intermediary we use to interface with reality but it is imperfect: it uses eyes and ears, languages and words, expression and cultural habits, ink and paper.
For instance, I think it is true that women cannot do all than men can and men cannot do all that women can. It sounds terrifyingly horrible but because it's too hard for me to express reality. I could continue and say the differences are irrelevant to the weight of their voices that should be equal, but then... why ? I used should here, and leave reality to start ideology. I could say that opportunities should be offered in equal proportion, but... how ? Are women/men in that context two discrete categories, or a gradient where people can choose what they want to be? And do they choose wisely at all time ? What is wise ? Is it constant in all time and place ? Should we distinguish men and women ? Should we ignore what makes them different ?
And now, I trapped the AI: it can't care, it doesn't have a daughter, a mom, a wife and a sister like I do so it will not overcompensate patriarchy like I do, maybe, while I really want my daughter to prevail maybe at the cost of someone else's son, so my fair is not their fair...
Searching for truth is important, but once you've reached it, all you have is ink on a paper transcribing in a grammar and syntax what your poor eyes and weak ears perceived of reality, or at least what your ever aging brain remembers of it. It is no different for the AI, limited just as us, but not exactly overlapping the same flaws.