I dunno, wasn't Saint Augustine "lord make me good, but not yet" pretty much admitting that our heroes have feet of clay, and yet they function as educators and leaders.
If you modulate the training set through externally derived axioms of good and bad, can't you train an AI on objectively naughty data to recognise the anti-set of good behaviour?
On the extreme end of this is - What one side considered a terrorist, the other considers them a liberating warrior.
That is easy at that scale but the finer grain you go on morals the more fuzzy it gets and the more push back you will get from all directions. This is how you get things like the 1946 Obscenity threshold of "I know it when I see it".
Many saw the 2008 crash as a major issue that destroyed the lives of millions of people, some hard line environmentalists saw it as the single great decrease in consumption we have ever had.
If AI is judging us, it's either based on data encoding our existing behaviors. Or it's crudely manipulated data by biased hubris filled megalomaniacs. The rest of humanity doesn't really have/isn't permitted much of a say in the matter.
My argument would be this is both likely and inevitable where there AI is used as a judge. OpenAI, for example, has material explaining how they prune and curate their datasets to alter it's biases.