Microsoft is aware of the downsides of false positives. In this case California police officers would use facial recognition on individuals, but Microsoft recognized that the false positives could disproportionately target women and minorities. That could result in massive backlash against the company.
So they are calling for greater regulation of AI-related tech, considering the human rights issues. No mention of what "regulation" could mean here.
A company can make moral decisions, and the fiction that lawsuits are likely or even plausible is almost entirely mythical.
Cigarettes are considered harmful to everyone, you don't magically get the ability to resist addictive, harmful substances when you turn 21, yet we ban sale of them to minors - for similar reasons.
These are the perverse incentives when the standard is "equal treatment under the law", or "equal protection under the law". If they can find some way to treat everyone in a crappy fashion, the law doesn't have a problem with that. The successful class actions tend to come only when it can be shown that you are only treating some people crappy.
That would be fairly easy to show in this case. Simply have random whites and blacks stand in front of a camera running MS software and compare the accuracy rates. (Come to think of it, a demo like that would be pretty powerful in a courtroom too. Maybe there is more to this than just PR MS is worried about?)
EDIT:
I just realized how bad that sounded. To be clear, I'm not saying we should treat some people unfairly. I'm saying the law doesn't care whether we treat people fairly or unfairly, so long as they all get treated the same. I was positing that maybe it would be better if we could try to make the law incentivize treating everyone fairly.
Nothing has ever had nor will ever have zero chance of false positives. Even DNA evidence can have false positives. Law enforcement, the justice system, and society overall has always had and will always have (highly imperfect) mechanisms to deal with nonzero false positive rates.
Therefore, it is of course important that the false positive rate not be higher than average for a subgroup, as they will be disproportionately affected, since our systems will be set up to deal with the lower average false positive rate.
Longer form:
Algorithms 'discriminate' (as in differentiate) because that is exactly the job they are tasked with. Is this a picture of a person on list A or not? Is this a picture of criminal activity X? They discriminate on a large number of, often hidden, unknown or not understood features in the data.
In many countries the laws dictate that some features such as race, religion, sexual orientation, ... are protected, as in not legally allowed to be used in differentiation. (Take note that in many countries certain national security/safety related organizations are exempt from certain regulations).
The models that are used in facial recognition rely (in part) on 'sub-symbolic' probabilistic feedback systems, that in many cases defy post rationalization: we do not have a convincing or specific 'story' about how 'the machine' decides in each cases. This means we can not deductively prove that the above mentioned 'illegal' forms of discrimination were not used (note that it is not sufficient to show that e.g. 'race' or 'gender' was not explicitly used as a feature in the input, as it could be strongly correlated with other inputs or derivations thereof (e.g. type of shampoo bought, zip-code, food preference, ...)
So we rely on things like testing post training deployed model to 'vet' the systems aren't biased in the ways we by law and regulation have deemed that they should not be. We test whether the output distribution shifts when we only feed in males vs a mixed gender test set etc.
In practice this means that in compliance testing we replace deductive reasoning with correlation. We accept that this will yield false positives, but this choice is partly due to technical limitations (apart from a few well published cases, understanding and explaining how e.g. a deep learning derived model actually 'works' in a rational synoptic way is still beyond us), and part due to ideological stances we have come to.
So, yes, we choose to accept false positives that are presented as evenly distributed across specific protected groups or features, while not 'making a fuzz' over others. These are inherently cultural, political, moral and empathic decisions, not 'logical' ones.
1) "The algorithm has a 50% false positive rate over everyone" -> awesome, no problem!
2) "The algorithm has a 50% false positive rate of people of color, and a .01% false positive rate over everyone else." -> yikes, we can't use this thing!