> Same goes for other basic things like women getting lighter sentences, minorities harsher sentences, etc.

The problem is, "AI" today is glorified pattern matching. So, it learns what "should be" based on the current state. In other words because the pattern exists it will learn precisely that "women should get lighter sentences and minorities should get harsher ones"

Not only that, they are so good at pattern matching that you don't even need to provide the gender or race for them to identify those as a parameter.

It would be great if we could train an AI to eliminate bias, but as it is they are training to reinforce bias that already exists.

Until we get natural language understanding and context aware AI, using AI for sentencing is a terrible idea.