Here is a somewhat modified situation. Say you use a machine learning algorithm to recommend if a person currently accused of a crime should be allowed bail. The fitness function should be to maximize the number of people who are allowed bail and minimize the number of people who miss their court date.

Say that the model allows X% of people to have bail and makes sure only Y% fail to show up.

The model then adds race as an input to the model. This improves the model so that it allows X+5% of people to have bail and makes sure that only Y/2% fail to show up. It also has the effect that it increases the chance that a black person is denied bail and increases the chance that a white person is allowed bail.

Do you think the inclusion of race is bias? Should race be removed from the input to the model?