The question then becomes what is the threshold to open an investigation? An AI represents reality better than a human, but also reminds us that certain classes and races of people are more likely to X Y and Z.
If reality is racist, and there is no such thing as equality or equity, is it ever ok to use an AI for any task at all?
Even while not feeding race data to the AI, it will still be racist because all datapoints should lead to an accurate representation of reality, and there is no such thing as equality or equity. In this example, even without race data, the AI should STILL come to the same conclusions. In fact, I doubt it knew the skin color or ethnicity of the names of the people it was analyzing.
Where the Dutch fucked up was jumping to conclusions and labelling without cause or investigation 20,000 people as fraudsters.
BUT, EVEN IF THEY CONDUCTED INVESTIGATIONS ON ALL 20,000, and they were guilty, does that make it any more right or less wrong?
I posit no. It's still illegal and wrong.
AI can only represent reality, reality reflects uncomfortable truths. AI has to be racist, or it isn't doing its job. The world is not equitable, all races have difference cultures and values and work ethics.
Which I posit means that even if the AI is correct, and even if those 20,000 committed fraud, its still an illegal prosecution as there was never a legitimate predicate to investigate in the first place.
AI is racist, even if it represents reality, and therefore cannot be used to initiate an investigation.
To me, a more interesting question is whether AI should be allowed to assist police in investigations, as even with a predicate for opening an investigation, the AI is still racist/sexist/classiest etc, and is going to help put people in jail based on racist classest sexist biases.
An example is a middle class white man that shares few traits with the average criminal, but is nonetheless a criminal. VS a black low income man that shares many traits with other criminals.
In this example, if an investigative predicate was found for both, and an AI was used as an assistant to the police, the police would be looking at the low income guy with more scrutiny as they would believe that a more likely case to win. Theoretically. Even though we know them to be guilty, the police don't know that, and the AI don't know that, but the AI is allowing for more scrutiny and conviction against one of the two guilty parties.
TLDR; AI is fucked. Shut it all down.