>Reached by The Verge, an Amazon spokesperson attributed the results to poor calibration. The ACLU’s tests were performed using Rekognition’s default confidence threshold of 80 percent — but Amazon says it recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences.
Presented without real comment on my part.
That's not in https://aws.amazon.com/rekognition/faqs/ - while that threshold may minimize false positives, I'm curious if it was in any documentation that the ACLU saw. Or, to the bigger point, is it in any public documentation?
They were slow to let go of phrenology as well.
It can say exactly what the Amazon response to the Verge dismissing the ACLU test should, and, in fact, it should say that if Amazon really does have such critical, specific domain-specific recommended settings, rather than it being something made up off the cuff to mitigate bad press.
With that said, it's possible that no amount of clear guidance would have mattered here. The ACLU isn't exactly asking for better docs.
If you're a researcher making a deliberate choice to use the defaults to simulate a random police officer with no real understanding of their tools, then say as much. And compare with more cautious results.
> A confidence score is a number between 0 and 100 that indicates the probability that a given prediction is correct. In the tropical beach example, if the object and scene detection process returns a confidence score of 99 for the label ‘Water’ and 35 for the label ‘Palm Tree’, then it is more likely that the image contains water but not a palm tree.
> Applications that are very sensitive to detection errors (false positives) should discard results associated with confidence scores below a certain threshold. The optimum threshold depends on the application. In many cases, you will get the best user experience by setting minimum confidence values higher than the default value.
Which I am interpreting as similar to your question, but perhaps should not be understood as a statistical probability.
That’s not how these numbers fit together. Using rekognition, you would create a collection of faces based off your image corpus and then ask the service if a provided image has a match against anything in your collection. Lowering your confidence threshold, invariant of your sample size, is by definition going to increase your false positive rate while lowering your false negative rate.
I think it would be worthwhile to compare a mixed test set and see false positives/negatives at various confidence thresholds, because the fear (IMO) is that law enforcement will minimize false negatives at the expense of a huge increase in false positives.
They should rerun this at 95% confidence and see how many mismatches they get, I suspect it will be much lower than the 6.4% they have at 80% confidence, thought it might cost more than the $12.33 they were willing to invest in this hitpiece against rekognition.