The worst I've heard of along these lines currently in practice is predictive policing [0]:
First the Sheriff’s Office generates lists of people it considers likely to break the law, based on arrest histories, unspecified intelligence and arbitrary decisions by police analysts.
Then it sends deputies to find and interrogate anyone whose name appears, often without probable cause, a search warrant or evidence of a specific crime.
They swarm homes in the middle of the night, waking families and embarrassing people in front of their neighbors. They write tickets for missing mailbox numbers and overgrown grass, saddling residents with court dates and fines. They come again and again, making arrests for any reason they can.
One former deputy described the directive like this: “Make their lives miserable until they move or sue.”
It seems that police departments are happy to buy tools for stuff like this, and private companies are happy to make a buck. But are these systems vetted by anyone on behalf of the public? Some like sting rays and breathalyzers have been hidden behind non-disclosure agreements, kept out of open court, etc.
Other than the above I've also heard of computer systems in NJ advising judges whether a suspect is a risk of not appearing in court (as they are reforming the bail system). I wondered if the criteria is published, how it reviewed and modified, etc.
I'm just concerned about the increasing influence of hidden algorithms on our society, and very concerned in general that the government is going to hook all of its databases together and do more of this, a la Chinese social scores, etc.
[0] https://www.techdirt.com/articles/20200907/12212945257/flori...