Why do we have due process? One key reason is that it gives people the opportunity to be heard. One could argue that being heard by an AI is no different from being heard by a human, just more efficient.
But why do people want the opportunity to be heard? It’s partly the obvious, to have a chance to defend oneself against unjust exercises of power, and of course against simple error. But it’s also so that one can feel heard and not powerless. If the exercise of justice requires either brutal force or broad consent, giving people the feeling of being heard and able to defend themselves encourages broad consent.
Being heard by an AI then has a brutal defect, it doesn’t make people feel heard. A big part of this may come from the idea that an AI cannot be held accountable if it is wrong or if it is acting unfairly.
Justice, then, becomes a force of nature. I think we like to pretend justice is a force of nature anyway, but it’s really not. It’s man-made.
This is a hypothesis.
I would say that the consumers of now-unsexed "AI" sex-chat-bots (Replika) felt differently. So there are actually people who feel heard talking to an AI. Who knows, if it gets good enough maybe more of us would feel that way.
eg AI will analyze stock trades for the SEC and surface likely insider trading. Pretty sure they already use tools like Palantir to do exactly this, it's just that advanced AI will supercharge this even further.
Eh, this is problematic for a number of reasons that need addressed when adding any component that can increase the workload for said humans. This will cause people to take shortcuts that commonly lead to groups that are less able to represent and defend themselves legally taking the brunt of the prosecutions.