Humans shouldn't use simple heuristics like this to make such serious accusations either way.
The problem with computer systems flagging things is they are taken as truth.
Another comment on another post perfectly states this human behavior: https://news.ycombinator.com/item?id=39118716
Wouldn't it need additional data, such as actual proof, to become an allegation or even a claim or charge?
Yes but that hasn't stopped people before.
A false positive rate of 1/10000 would be almost the worst case in fact: truthy enough that people believe that it works, while still creating a vast absolute number of false positives.
What the computer surfaces as an accusation becomes fact by the users of the computer.
Even when the consequences are serious, this happens, an example from yesterday: https://www.nbcnews.com/news/us-news/man-says-ai-facial-reco...
Why should we assume this would go any differently when the consequences are not as serious?